While that's an insanely cool problem to work on, even better was that Nvidia and Dell sponsored the event and gave each team a Dell Pro Max workstation equipped with an Nvidia GB10 Blackwell GPU to use for the day1. For context, this is a $6k+ beast of a machine with 128GB of VRAM that can easily run some of the best open models out today. Which is also how I found myself up at 11 pm the night before, downloading a bunch of models onto my laptop, as the venue Wi-Fi had no chance of pulling files that big. So whatever we wanted to run had to come with us, and hopefully at least one of them would behave on a GPU I had never actually touched.

Even cooler… something special must be happening in our industry if a hundred people, ranging from college students to seasoned professionals, decided to give up their Sunday to build business software for the love of the game.

But does this AI-induced FOMO mean you need to be jumping onto the local AI hardware bandwagon as well?

In short… no. For most people, this is a massive waste of money (GPUs cost about the same as used cars). But everyone I know, including myself, who has put serious money into this space, sees the same major opportunity opening up and is positioning themselves to be first movers.

We're still very early for local AI, but it's wild that by the end of a hackathon, I had a team of AI agents running on that single GPU on my desk. Furthermore, that agentic team took a real issue from my open-source project, turned it into a tested pull request, and then attempted to fix its own failing build automatically. In this article, I highlight my own experiences and reasons why I'm all in on local AI, as well as share my current stack and how you can get started with serving local AI models.

Hackathon Case Study: AI Software Factories

So what did my hackathon team decide to build? The coveted AI factory to build software from a spec alone, as described by NVIDIA’s CEO Jensen Huang. While ambitious, our goal was to prove if the following end-to-end run is even possible:

1. Can we get a local AI model deployed to the GB10 and access it from our personal laptops?

2. Can we deploy a local agentic harness to coordinate a team of AI agents to run the entire software development lifecycle?

3. Can this local AI agentic team take a real-world issue from my open-source project, Petri, and create a PR with a code change?

4. This first PR will likely fail my CI/CD workflow; can it automatically detect the failure, attempt to resolve the issue, and push a new commit to the PR with a fix?

5. Will I accept this fully AI-generated code and have this change deployed to PyPI?

Looking at this list, I assumed the model system prompts for each agent would be the hardest part of our day. It was not. But we’ll get there, and at the end I’ll come back to all five of these questions as a hackathon scorecard.

Furthermore, in addition to all LLM calls being local, the hackathon rules also required us to use OpenClaw (AI harness) and OpenShell (governance, sandbox, and security). The GIFs below (generated via Claude Fable 5) summarize how this all works together.

So how can you stand up this build yourself? I’ll walk you through each of the above questions and even share the pull request this local AI agent submitted!

Serving Local AI Models

Thankfully, this wasn’t my first rodeo deploying local AI on NVIDIA GPUs, and thus we used a similar setup to my own local LLM server for the build.

From the pile of open weights I’d downloaded the night before, we ultimately decided to go with Qwen3.6-27B as it balanced performance with model speed on the hardware we were given for the day.

Hardware

This is going to be the main gate preventing most people from even considering local AI. Full stop, this is expensive, and you are most likely better off using a neo-cloud to run open-weight AI models on rented GPUs. Shane Morris’s article, You’re going to break this thing that costs like $9,000. I hope you made peace with your God before you did, perfectly sums up the current state of the space and who should and shouldn’t pursue local AI hardware.

That’s where we are. That’s this whole scene, right now. “Own your model” is going to be a weird hobbyist phase for a while, until the people out on the bleeding edge sand down all the rough spots and hand it to everyone else in a box that just works…

I broke a working system because I was playing with it, which is why you can’t (and shouldn’t) do this unless you’re committed to learning a lot while you fail…

So if that sounds fun to you: Cool. Genuinely. Have at it. Tinker. Break it. Fix it at 2am. If this gives you the same joy and frustration it gives me, dope. Go do it. 1 in 1,000 people will like this process.

If the above still doesn’t deter you, then you are about to have the time of your life genuinely building, failing, and learning on the bleeding edge. I highly suggest first using rented cloud GPUs to understand your use case and whether you even want to play with these models. Here are some questions you should ask yourself?

• What level of inference do you need for your tasks (e.g., Qwen 3.6 barely fits on my 64GB M1 Max MacBook Pro; Kimi 3 with frontier capabilities requires full server racks of 32 separate H100 GPUs, with one of those GPUs costing ~$30k each)?

• Are you just using it for a workstation and R&D, or are you planning to serve models to multiple people concurrently with a cluster of GPUs?

• Do your tasks require high inference (e.g., hard scientific questions), medium inference for long-running tasks (e.g., AI evaluations), or low inference at high concurrency (e.g., agentic swarms)?

• Do you plan to run only local models, or are you also trying to fine-tune AI models on your own machine?

• What security and governance requirements do you have for the work you are doing?

For my own business, I landed on a single-node RTX PRO 6000 Blackwell workstation pictured below, with the intention of:

• A) Doing R&D on serving local AI and running agentic tasks on it for highly regulated industries that can’t use frontier models (e.g., healthcare, law, etc.)

• B) For the RTX Pro 6000 specifically, fine-tune small models for specific tasks such as NVIDIA’s nemotron models.

That’s my main thesis right now, and where I think local AI will be huge—with all of my free time running experiments to fully understand this emerging opportunity.

Open Weights + vLLM + llama-swap

Once you have settled on hardware, you unfortunately can’t just download a model off Hugging Face and start using AI. That luxury is reserved for brew install --cask claude-code and why I pay $200/month for a Max plan. Oh, my sweet summer child, you have to:

1. Determine which model works on your hardware across the litany of configurations, fine-tunes, and quantizations that all have their own quirks for every machine (this is a mix of art and science).

2. Get the weights loaded onto the GPU (i.e., “pre-fill” phase) and hope you set the right model configuration flags for things like context size, tool use, and other model-specific quirks (I will use this word a lot).

3. Manage the KV Cache for the “decode” phase of output generation and keep track of prompt history.

4. Provide a way for your hardware to either maintain multiple models (if your VRAM permits) or manage swapping model weights on your GPU.

5. Make sure your hardware doesn’t spin towards self-destruction given your selected acceptable quirks.

All of this is exactly why I decided to go with a single GPU build, as managing the above gets exponentially harder once you have to use multiple GPUs in this crazy local AI memory dance. I could have had way more VRAM at a fraction of the price—like the badass CTO at my day job who amassed a bunch of used GPUs and is now running multiple Qwen 3.6 models on 512GB of VRAM—but I just knew my use case didn’t warrant this increase in complexity. I’m more interested in building local and secure agentic applications for highly regulated industries… I don’t want to become a hardware engineer.

With all that said, over the past few years we have seen massive open-source advancements that have made this move from deep-tech magic to something approachable, if you don’t mind mucking around in the terminal. In particular, the vLLM project from researchers at the Sky Computing Lab at UC Berkeley (now part of the Linux Foundation) has been a game changer and is the go-to LLM inference and serving engine for production builds.

Finally, I couple vLLM with this cool project, llama-swap, and now I can easily hot-swap models on demand. This space is moving so fast, but even more exhilarating is seeing all these friction points of deploying local AI are being solved by the open-source community.

Agentic Harnesses

Great, you have hardware and an LLM serving engine all ready to go! We now start getting into more familiar territory if you have spent any amount of time with coding agents like Claude Code, Codex, or Cursor. I have covered agentic harness in my earlier article linked below, so I won’t re-hash it… but things change when you move to local hardware.

The whole point of using local hardware is to decouple from the frontier AI labs and, more importantly, have sovereignty over your models, data, and IP. While Claude Code is the best harness I’ve ever used (as of today), using such creates a new dependency that won’t fly in regulated and/or sensitive environments (my specific use case). With that said, out of the open-source options, I am still trying to figure out the best setup, and I’m actively seeking advice from the community on this.

So far, I have ruled out OpenCode as its harness system prompts are way too heavy for local models, and in my comparison of using Qwen 3.6 via Claude Code and OpenCode for complex coding tasks, I was left severely unimpressed despite wanting to like OpenCode (this may very well be a user error). Thus, I’m now evaluating the two following options:

NemoClaw

• Essentially NVIDIA’s response to making OpenClaw safe enough to run in an enterprise setting.

• Under the hood, it’s OpenShell for sandboxing and governance plus agents like OpenClaw or Hermes agent.

Pi Agent Harness

• The most bare-bones harness, designed as a blank slate that anyone can build on top of for their specific use case.

• Being an open slate makes it perfect for hyper-specialized harnesses that are not encumbered by unneeded system prompts like OpenCode.

Right now I’m landing on Pi harness for specific agentic workflows (I’m actively implementing it into Petri) and NemoClaw + Herms Agent for my overall agentic operating system. Expect an article on this as I learn more from my own builds!

Tailscale + AI Gateways

At this point, you have everything you need to run AI on local hardware— which is pretty damn cool! But we can make it even better…

While my AI is local, I don’t need (nor want) to be local myself. This is why I use Tailscale to easily manage access to my LLM server from all of my devices, including my phone. On top of making my hardware accessible from anywhere, it also gives me end-to-end encryption, 2FA, and the ability to share access to my machine with collaborators.

For example, I have an executive assistant who I want using AI for many of our workflows. But my business doesn't have enough seats to qualify for a business account with Anthropic, which is how you get Anthropic’s Commercial Terms of Service and the Data Processing Addendum signed for data protections. Without those protections in place, my sensitive data isn't going into a frontier lab's products, so it stays on hardware we control. Thus, we are working towards upskilling her in AI with these DeepLearning.ai courses and giving her access to my LLM server to create her own agentic workflows to support my business.

In particular, I can’t expect my executive assistant to jump into the terminal and easily run AI workflows. I need to abstract away as much as possible so all she has to worry about is logging in with my business’s email domain and using a chat window. It’s on me to ensure the proper guardrails are in place (e.g., accepted tools, safe AI models, what domains the agent can touch, etc.). An AI gateway makes this possible, where I can have all LLM calls go through a central point that I can observe, manage, and ensure the LLM doesn’t behave in misaligned ways.

To be clear, this is 100% overkill for most people playing around with local AI (including my work), but it provides me a lot of insight into what friction points will emerge if a company seeks to responsibly use local AI hardware.

Putting It All Together

Nearly everything mentioned above (e.g., vLLM, OpenShell, etc.) had to be set up by us at the hackathon. It was fun, but we still hit a bunch of issues!

Remember that 11 pm model-downloading session? Turns out slow downloads were the least of the venue network’s problems. The museum had a super intense firewall and almost no one could properly SSH into their Dell GB10 machines! Thankfully, the hackathon organizers found a workaround that worked on my teammate Daniel’s machine but not my Mac. As the hackathon gods laughed at me, Daniel’s debugging found that our specific GB10 (out of 40) could ping all computers except mine, and that my Mac could find other GB10s except ours. ~laughing intensifies~

The workaround we found three hours later? We turned my phone into a hotspot, connected the GB10 and my Mac to that hotspot, and then added the GB10 to my Tailscale network so I could finally SSH into it with my Mac.

With SSH finally working, we set up the infra, uploaded the machine with our OpenClaw agent configurations, pointed the AI software factory at a real issue from Petri’s backlog, and let it rip. The agents broke the issue down, wrote the code and tests, and opened a PR against my repo… which promptly failed my CI/CD workflow (shoutout linting). Then, with zero input from us, the factory read the failure and pushed a fresh commit to its own PR. All of this was happening while we recorded our quick scrappy demo for the hackathon judges (submitted with only a few minutes to spare):

As for the code our AI software factory produced? I don’t plan to merge it into my project, but the whole hackathon showed me that it’s possible with local hardware and open models. That’s enough signal for me to commit another weekend iterating on this idea and hopefully make the code public.

So, scoring our five questions from the top of the article:

1. ✅ Local model on the GB10, accessible from our laptops? Yes (with a three-hour detour courtesy of the museum’s firewall).

2. ✅ Local agentic harness running the entire software development lifecycle? Yes.

3. ✅ Real issue from Petri turned into a PR with a code change? Yes — here’s the pull request.

4. ✅ Automatically detect the CI/CD failure and push a fix? Yes. Watching this one happen was the highlight of my day.

5. ❌ Will I accept this fully AI-generated code and deploy it to PyPI? No.

I call this a massive win doing all of this in about ~12 hours!

Closing Thoughts

Wow… we are living in the future. I’m not trying to stoke the FOMO flames to convince you to purchase what is essentially a used Honda Civic in the form of a GPU, but this is the most fun I’ve had building in years. With that said, the technology is still not there for most consumers and businesses to use local AI hardware… but you can see the momentum of where things are going and the impact it will have on so many industries.

Are you building with local AI? I would love to hear from you and learn!

Found my content interesting and want to learn more from me? I have four courses on LinkedIn Learning teaching you the fundamentals of data quality and data engineering best practices. With over 40K+ students and hundreds of five-star reviews for my hands-on coding courses, I’m confident you will learn some valuable data skills and have fun while doing it!

Feel free to check them out here:

• Data Engineering with dbt

• Data Quality: Core Concepts

• Data Quality: Analytics and Serving

• Data Quality: Transactions, Ingestions, and Storage

The Substack Post

Against Claudefishing

It’s getting harder to tell what’s real on the internet…

Read more

22 days ago · 6077 likes · 1297 comments · Chris Best

While I personally find this fascinating as a data quality nerd, I would be remiss to call out the wider context of such a decision. Who is the arbiter of truth in a world where written “prose” has been commoditized to a series of API calls? Such a question dêlves3 into philosophy and defining ethical AI use, which I feel requires perspectives from multiple disciplines. Thus, I share my views from a data quality lens and hope others can add to the conversation.

What is Pangram?

Thankfully, Substack didn’t try rolling out an internal feature for AI detection (i.e., the “trust us, bro” approach). While this doesn’t preclude scrutiny, having a third-party with open-source research45 behind their engine is a step in the right direction. According to Pangram’s article6 linked in Substack’s docs:

Pangram is a statistical model that makes predictions based on text alone, so its accuracy must also be tested statistically. That is what we do. We publish internal benchmarks for every Pangram model we release, as well as every time an AI lab updates their LLMs. We have also been tested and verified by independent third parties, such as teams at the University of Chicago and the University of Maryland.

Pangram

How does Pangram work?

We think it is important to be able to reliably identify AI-generated writing, so we built a detector that does it. Because the problem is so important, it’s only prudent to be transparent about how that works…

Read more

a month ago · 152 likes · 38 comments · Pangram

Thus, it comes down to the central question of data science: to what degree can we trust the results of a statistical and or machine learning model, and is this result useful to the business?

What Makes a Model Useful?

In my own data career, I have developed classifiers that made it into customer-facing products or powered critical business workflows. None of them had 100% accuracy.

Accuracy improvements require exponential effort as you get closer to 100%, and thus businesses need to make tradeoffs. For example, say you had a vendetta against Big Toothpaste, and you wanted to validate if “9 out of 10 dentists” truly recommend a particular brand. If you were fully committed to the cause, you would spend substantial amounts of money and resources on surveying every dentist in the world on their recommendations. And then reality sets in… you listen to the statisticians and use sampling methods as a reasonable and sane tradeoff. This is the crux of data quality.

My favorite piece of research in the data quality space, Beyond Accuracy: What Data Quality Means to Data Consumers by Drs. Richard Wang and Diane Strong, defined this as being “…fit for use by data consumers.” To me, this definition from the 90s is perfect, and has been my guiding mental model for evaluating AI products.

Thus, as a consumer of Pangram’s AI detection outputs via Substack’s platform, is the underlying data fit for use? For me personally, it absolutely is, as I’m so desperately tired of the AI drivel masquerading as novel insight on posts and comments throughout the internet. With that said, I recognize I’m being presented information that could be wrong, and thus I use my judgment.

With that said, my curiosity still gets the best of me…

Is Pangram Accurate On This Article?

I’m going to apologize upfront… The second half of this article will be nothing but pure AI slop. BUT FOR SCIENCE!

This section and every section before in this article was written by my own human hands with intermittent spell check from Grammarly. How would the Pangram AI detection model grade this article if the section below was:

1. A summary of the Pangram 3.3 model card written fully by Anthropic’s Fabel 5.

2. Was the same character length as the previous half of the article (3,937 characters).

I personally expect it to either show 50/50 human and AI-generated, or 50% AI-generated and a mix of human-AI-assisted (i.e., Grammarly). I’ll make sure to add an edit after I publish and share a screenshot of the results!

Edit:

THE SECTION BELOW IS AI-GENERATED

# Pangram 3.3 Model Card: A Summary

Pangram Labs has released Pangram 3.3, the latest version of its AI-generated-text detector, and the accompanying model card offers an unusually candid look at what changed, what improved, and where the model still struggles. Released on May 13, 2026 as the successor to Pangram 3.2, the model is built on the company’s EditLens architecture, presented at ICLR 2026, which frames detection not as a binary human-or-AI question but as a measure of how much AI intervention a text contains.

## What’s New

Pangram 3.3 was optimized for a lower false negative rate on the newest generation of large language models, including Claude 4.7 and GPT 5.4 and later. Internal evaluations show a 3x improvement in detecting GPT-5.5 Pro text and more than a 4x improvement on Claude Opus 4.7 compared to Pangram 3.2.

The model also makes significant gains against evasion attempts. It catches twice as many commercially humanized texts as its predecessor and shows a 3x improvement on Pangram’s internal adversarial dataset, which consists of LLM outputs where users explicitly prompted the model to evade detection.

Long-form content is another focus. Pangram 3.2 occasionally classified AI-generated documents over 2,000 words as “mixed,” mislabeling later segments as human. Version 3.3 significantly reduces this error, classifying long synthetic texts as fully AI far more consistently.

## Fewer False Positives

Pangram states it will never release a model that improves recall at the cost of misclassifying more human writing. True to that policy, 3.3 reduces the false positive rate on non-native English (ESL) writing while the overall false positive rate actually decreased, thanks partly to improvements in challenging domains like poetry. The card cites a 0.01% false positive rate in common use cases such as creative writing.

## How It Works

The EditLens architecture uses bucket-based classification: a transformer emits logits across levels of AI pervasiveness, decoded into four categories — Human-Written, Lightly AI Assisted, Moderately AI Assisted, and Fully AI-Generated. These collapse into a single ai_assistance_score between 0 and 1, which is now normalized to be more easily interpretable.

Training data and supported languages are unchanged from 3.2: a human corpus of long-form prose spanning essays, creative writing, reviews, books, Wikipedia, news, scientific papers, and web text, with support for 22 languages. The minimum input length remains 50 words, giving the model enough context to make a trustworthy prediction.

## Evaluation Approach

The card distinguishes three evaluation types: in-domain test sets, out-of-domain evaluation on completely held-out sources and domains to measure generalization, and external benchmarks — which Pangram cautions “should not be trusted as a current measure once released, as benchmarks can be trivially trained on.”

## Limitations

Pangram 3.3 is intended for long-form writing samples in complete sentences. Bullet point lists, technical manuals, tables of contents, reference sections, templated writing, and dense mathematical equations are more susceptible to false positives. For best results, Pangram recommends removing human-written instructions, headers, footers, and other extraneous formatting before checking a text. One honest disclosure: 3.3 shows a slight uptick in false positives on human text passed through Google Translate, an issue the team is actively working on.

## Version History

Pangram 3.3.1 (May 15, 2026) kept the same underlying model but improved the segmentation algorithm for documents over 450 words. Pangram 3.3.2 (May 18, 2026) was a small bugfix release affecting fewer than 3% of predictions.

## Bottom Line

Pangram 3.3 is a meaningful step forward in AI text detection: better recall on frontier models, stronger resistance to humanizers, and fewer false positives — all with transparency about its remaining weaknesses.

It’s been a while since I last posted on here, but for good reason, as I’ve spent the past few years crossing off a bucket list item of writing an O’Reilly Book on the topic of Data Contracts. The book was published in November 2025, so what have I been doing for the past five months?

Spending every minute I could spare using Claude Code until my needs eventually outgrew it (more on that soon).

If you are a techie like me, then I imagine the past five months have been similarly intense. While I’ve been using AI-assisted coding tools since the copy-paste-from-chat-window days, last November felt like a major inflection point where AI went from “Nice auto-complete!” to “This just wrote the entire script better than me in one minute…”

It’s jarring at first, as you question the value you now bring to your career, so you dig deeper to understand. This is also why you see so many people building intensely right now. It hit me the most when my friends outside the intense startup grind started sharing the crazy hours that matched my 80+ hours a week. While I can’t predict the future, going deep on these tools has only shown me how important software and data foundations are. We are not going anywhere, but our work in software and data is drastically changing.

This article details the key areas I think data practitioners should focus on to evolve toward agentic systems and proposes the micro-orchestrator pattern as an abstraction that I believe will prove useful as agentic workflows grow. In addition, I provide a real-world example of a micro-orchestrator through an open-source Python package I created and break down key considerations for building your own!

Subscribe now

Part I: Emerging Agentic Patterns - An Abridged History

I first heard about AI Agents in early 2023 when I interviewed Jazmia Henry about her work as an AI researcher—back then, she was on the Microsoft team that ultimately became part of OpenAI. It’s wild how accurate her quote is today:

So that's the biggest challenge [of AI agents is] having a goal and that goal being as close to the actual problem, especially if you’re gonna deploy it. And then the second thing is finding that darn data and making it as close as possible to the actual world so your agents actually learn something of value and not fail.”

Unbeknownst to me, this single interview altered my course in data and made me hyper-aware of the data industry's changing landscape. Back then, the wider data industry was still obsessed with OpenAI’s ChatGPT. She was simply talking about the future we are experiencing today. Since this interview, there have been three specific architecture patterns that have emerged to make these powerful AI models useful beyond chat windows:

1. The AI agents themselves, and specifically the ability for LLMs to use tools for long-horizon task completion.

2. Agentic harnesses, such as Claude Code, enabled LLMs to manage memory, use tools, and, overall, create the winning UX for agentic coding.

3. The emerging category of agentic orchestrators for reliably completing repeated workflows at scale.

These three changes are the inflection point that moved agents from a cool experiment to fundamentally reshaping how software is built and every assumption around it.

AI Agents

The second article that shifted my data career was Chip Huyen’s Agents blog, which was adapted from her O’Reilly book chapter on the topic. Specifically, she provided this insight:

A system doesn’t need access to external tools to be an agent. However, without external tools, the agent’s capabilities would be limited. By itself, a model can typically perform one action—an LLM can generate text and an image generator can generate images. External tools make an agent vastly more capable.

At the time, most AI engineers and researchers I knew advised moving away from chat windows and using LLMs via their APIs for greater control. Claude Code merely made best practices accessible to the wider developer community.

I highly recommend reading Chip’s article, as it covers AI Agents better than I could in this short paragraph. But our main takeaway is that we must think about AI Agents as “LLMs + External Tools” as the core primative for agentic harnesses and orchestrators.

Agentic Harnesses

As the utility for LLMs grew, so did our demand for long-running tasks that were not tethered to a chat window. Thus, to keep our robot friends running continuously, the industry has focused on leveraging agentic loops (i.e., Observe → Plan → Generate → Verify) via what are now called harnesses. Yet, spend any amount of time with AI agents, and you quickly learn that they are eager little geniuses that will do exactly what you say— regardless of whether what you said is your actual intent. Their capacity for synthesizing knowledge is only exceeded by their capacity to wreak havoc if you give it too much room (i.e., the siren call of --dangerously-skip-permissions).

While many like to point to the latest frontier models and their ability to wreck the curve on Humanity’s Last Exam, the sudden jump in agent utility points back to the harnesses themselves. Specifically, the harness provides human operators with a mental buffer against the deluge of decisions an agent must make and enables self-checking agentic loops that request a decision only after a goal has been met.

Agentic Orchestrators

So what’s the next obvious step for AI agents if you are a token-maxxing degen like myself? Orchestrators! My first introduction to this world came from using Claude’s Agentic Teams feature and reading Steve Yegge’s article, Welcome to Gas Town, where the popular social media image of Stages of Dev Evolution stems from:

Core to this process of orchestrating agents is optimizing their context windows to constrained subtasks within a larger workflow. If you have used the Plan feature of Claude Code, you have seen this in action, where it takes a request, determines the goals for completion, and then uses sub-agents to do the work in parallel. Orchestrators take this approach and then ask how much we can scale it horizontally while still maintaining reliable outputs.

I believe orchestrators will represent a meaningful amount of development work as agents become increasingly embedded in our workflows. Primarily because much of the conversation of value and revenue is moving towards an emphasis on “outcomes completed” instead of “work completed.” Thus, domain-specific yet repeatable workflows that were often not targets for automation—due to significant context-specific edge cases—are now becoming possible for organizations. This is also why you see so many non-technical subject matter experts (SME) driving insane value with vibe coding.

The problem is that this demographic of Claude Code users isn't exposed to what’s actually needed to move from a prototype they use only themselves to actual production software businesses depend on. The latter is why developers are safe in this new AI world for now, but this doesn’t discredit the fact that these domain-specific vibe coders are identifying valuable agentic use cases. Similar to developers supporting data science teams by turning their Jupyter Notebook prototypes into production software, the same is now happening for non-technical SMEs.

Part II: A Case For Micro-Orchestrators

To be clear, there are already agentic orchestration frameworks, such as LangChain, but they are heavy and akin to a company adopting tooling like Airflow. This is why you have seen the proliferation of agentic skills, which provide a lightweight alternative that enables repeated workflows by simply writing or importing Markdown files with plain English instructions.

Markdown Development Is Not Enough

Despite how powerful they are, agentic skills have an Achilles' heel: they are unreliable without constant maintenance effort or guardrails. Your AI agent first needs to know the skill exists, understand what context to trigger the skill, and even with that context, the agent may still not trigger the skill. Even if you put in the work to achieve reliable skill utilization, the constant changes to the underlying frontier models’ weights and system prompts can cause skills to drift in unexpected ways.

With generalizable orchestration frameworks being too heavy and agentic skills being too fragile, there is a need for what I’m calling micro-orchestrators.

Abstractions Move From Software to Taste

As I mentioned before, much of the value in agentic AI deployments lies with subject-matter experts who are closest to the most valuable problems the business is solving. Furthermore, if it were just a technical problem, these workflows would already have been automated—they are not, due to context-specific edge cases that only SMEs can handle. AI agents have changed this assumption, as their core strength lies in retrieving specific information, synthesizing patterns within their active context window, and then taking action based on inputs and/or responses.

For example, in a previous data science role, I developed data quality algorithms for unstructured electronic health record data and thus needed to collaborate with subject-matter experts. Specifically, physicians with sub-specialties (e.g., ophthalmology) on our staff would validate my assumptions and ensure clinical accuracy. Now imagine physicians who are no longer encumbered by a lack of technical skills and can build tools that map directly to the problems they see in their own clinics.

Coupled with plummeting software implementation costs, businesses are rushing to deploy their own AI agents to handle work directly, rather than relying on users to log in to a SaaS tool. Thus, the bottleneck now becomes taste, as OpenAI’s co-founder succinctly puts it:

Now, some may be saying, “Well, isn’t that exactly what the job has always been?” To which I respond, yes, but now the work is compressed in ways I argue the wider industry is currently struggling to handle (no one is being spared; Examples: Amazon, GitHub, Meta, Anthropic, etc.). Having your end user be both humans and agents is completely new for most people and companies.

Another way to put it, the same way you offload development to open-source packages is now being expanded to offloading taste. We have moved from “now I don’t have to build it” to “now I don’t have to detail my thinking for agents.”

Thus, offloading cognitive demands for taste in the context of workflow completion is the ultimate goal of micro-orchestrators.

Part III: Building Your Own Micro-Orchestrators

I have yapped A LOT, but one of the main lessons I’ve learned in this new AI world is that talk is cheap… ideas need to be accompanied with something built. So I created this fun project to bring this idea to life (and pay homage to my favorite video game). To my shock, there have been over 3.5k downloads on PyPi.

Case Study - Petri

Specifically, when I work with agents, I spend considerable time curating context (internal documents, prompts, sources of information, etc.). Thus, I aimed to automate that process and then open-sourced it as Petri.

You provide a claim, such as “open-source AI models will reach frontier lab levels in a year,” and the orchestrator will break it down into first principles, find recent and relevant citations online, evaluate the quality of the citations, and then build an argument for or against the claim with various nuances. What’s wild is that I also got this to fully work locally with OpenCode and Qwen 3.6-35B-A3B-GGUF on my MacBook Pro (M1 Max, 64GB)!

You can download it directly via uv pip install petri-grow or by prompting the following into your agentic harness of choice:

You can also just watch a fun demo video here… yes, Petri has video game background music.

While this is great and all for my own workflows, what about your specific work? The next few sections will break down key considerations for building your own micro-orchestrator.

State Machines - The Backbone of Agentic Workflows

Earlier, I stated that agentic skills are powerful but fragile. I learned this firsthand while building Petri, where the first iteration was handled fully through agentic skill markdown files. While it looked cool having my terminal buzz with agents, the massive burn of paid tokens left a mess of “research thoughts” that ultimately didn’t follow the complex workflow. Specifically, the agents were more eager to complete the task than to complete it correctly, and optimized for the quickest, ideal success path.

My second iteration still leveraged all agentic skills but added human review checks at each decision point for me to verify, with agents creating GitHub issues that I would comment on for feedback. This actually worked and validated that it was possible, but this approach was not feasible beyond this iteration. In particular, I spent over 10 hours reviewing and commenting on 100+ AI-generated GitHub issues. I learned A LOT about how agents operate across various logic scenarios, but I wouldn’t want to do it again. It became clear that I need to spend more time determining exactly where an agent should be used and optimizing my workflow for deterministic mechanical processes whenever possible.

Thus, I asked myself how I could establish a workflow that followed a deterministic path while allowing non-deterministic outcomes. I quickly realized I had covered these exact scenarios in my grad school public health modeling classes, specifically that I could use state machines to represent the execution steps of probabilistic events. What resulted was the following state machine for Petri (please note you can see a detailed version in the docs):

I cannot emphasize enough how big this is. If we can create a state machine that an agent follows, then we can also predict how changes to this state machine will result in changes in agentic behaviors and outputs, given that it’s all math and probabilities. In public health, this is essentially how epidemiologists can perform sensitivity analyses of various disease-spread scenarios.

Why Agentic Workflows are a Distributed Computing Problem

Another surprising learning from building Petri was how much agentic engineering is essentially data engineering. Again, we want to scale this work horizontally, so instead of having 30+ developers on a platform, I need to build for the same scenario across all the agents I oversee. Thus, I reframed the requirements as a data engineering problem rather than an AI one:

1. You break a complex workflow into individual tasks to keep the context window small and contained to only the task that needs to be completed.

2. These tasks have components that are sequential or can be done concurrently, but we need to assume that ALL happen independently.

3. Given that each agent and task are independent, the deployment of an agent (not its runtime) in a long-running workflow must be idempotent (deployment, not output).

4. Given that any independent task can fail at any time (hooray, non-determinism), the workflow must gracefully handle failures and retry the task.

5. Each state of the task, whether completed or failed, needs to be added to an append-only log that is edited programmatically (not by agents).

This is essentially a data pipeline, where the “data” we are moving and preparing for our end-user (i.e., the AI agent) is context. Even better, despite AI agents being bleeding-edge, we can leverage tried-and-true data architecture patterns to provide “shortcuts” for improving their reliability and utility. In particular, distributed computing problems map extremely well to the orchestration of AI agents, which also points to the use of event-driven architectures.

Event Sourcing and Immutable Logs

Writing the Data Contract book made me quite familiar with event sourcing, as its section required the most research and stress, given my primary experience in batch systems. While most people point to Martin Kleppmann’s book, Designing Data-Intensive Applications, as the quintessential book for our field… I posit that his magnum opus (at least to me) is his Orielly Report published a year before, Making Sense of Stream Processing.

One of his main arguments for adopting event sourcing was the use of an immutable log that all systems read from to maintain the proper state for their respective tasks. In particular, each unique system has its own specific business context and use case, which may overlap but ultimately serve different purposes, even though they are derived from the same data. The figures below illustrate how that maps extremely well to agent orchestration. In particular, since AI agents are stateless, we can optimize their memory and context window by feeding the latest event from our event stream as input to their next task and relevant triggers.

For example, in Petri, the Lead Agent reads the event stream to determine the last completed task. If the log shows that the Research Agent wrote a failed task (e.g., didn’t have the correct permission to view a website), the Lead Agent will flag a GitHub issue (or even in the main harness chat window) that a permission is needed. Furthermore, all other concurrent agents will recognize, from the event stream's immutable log, that this task is in the stalled stage of the state machine and proceed to other tasks until the queue is complete or all remaining tasks are stalled. This is powerful because now we don’t need to waste context on ensuring the agent is aware of an unrelated task— instead, it sees the current state, its options given the task, and a source of truth it can always reference to ensure it’s reaching its proper goal.

In other words, the immutable log from event sourcing enables full audits of agents (when tied to observability) and allows an agent to retain context and replay a workflow if a failure occurs and a new independent agent needs to be re-run. By applying data architecture best practices, we unlocked significant reliability and memory capacity for our agentic workflow.

Agentic Hierarchies for Context Management

One area of agent orchestrators that is both fascinating and where I think Petri has a lot of room for experimentation is how you organize agents and the boundaries of their communication with each other and or with human operators. This is also where I think many executives’ heads are when they think about how AI transforms work. For example, Jack Dorsey (CEO of Block) and Roelof Botha wrote an exceptional article, "From Hierarchy to Intelligence," drawing parallels between AI agents in the workforce and shifts in communication across human organizations throughout history. In addition, some of my former Humu colleagues are now leading rigorous applied research efforts on the topic at Atlassian’s Teamwork Lab.

For Petri, I opted for a three-level hierarchy, as illustrated in the figure below, composed of a main agentic harness agent, a lead agent for a given queued task, and under the lead agent, a set of specialized subagents for each stage in the state machine.

Furthermore, only the main Harness Agent is aware of all tasks and can dispatch Lead Agents for each queued task, with the only provided context being the task itself. The Lead Agent has three key roles:

1. Dispatch specialized sub-agents based on the state machine triggers.

2. Synthesize the sub-agents' outputs and make a decision.

3. Given that decision and context, trigger the event logging script.

This clear separation of information flow results in agentic workflows that run for hours, rarely have a context window exceeding 100k tokens (i.e., no context rot or compactions), and ensure multiple context perspectives are presented for synthesis. The latter point is a little less obvious, but most people are familiar with how syncopathic LLMs are and their affinity for recency bias (e.g., going overly positive or negative, based on the most recent prompt). By leaning into that constraint, you can have an ensemble of agents given different angles (e.g., for Petri, making a case for and against a piece of evidence), and then have the Lead Agent synthesize all perspectives into a balanced response.

As I said earlier, there is ample room here to optimize your micro-orchestrator and showcase your “taste” for these workflows. For example, in Petri, many of the improvements I’m working on center on the logic of how agents break down claims into specific units of logic and research questions—a challenging but fun problem.

The Data of Agentic Harnesses

Finally, no improvements to your micro-orchestrator can occur if you don’t actually capture its logs and, more importantly, read the trajectories of the agents (i.e., the turn-by-turn inputs and outputs of an LLM within a harness and its accompanying metadata). Thankfully, many agentic harnesses come with pre-built logging that is emitted as streamed JSON (e.g., for Claude Code CLI, it’s --claude -p "query" --output-format json-stream). This is a great starting point, as you can locate the local files and read them for yourself.

You can take this a step further by passing these logs into an open-source agentic observability tool such as Langfuse or Signoz, and extending them to the orchestrator itself via the OpenTelemetry (OTel) specification. What’s most exciting is that the wider industry is standardizing on OTel, making it easy to integrate agent tracing and observability across multiple tools. I have implemented this in other private projects, and it’s downright terrifying when you actually dig into the trajectories and see how the agents’ output resulted from not properly following a set of instructions, and instead creating a mock of what was expected. This was the moment I knew observability was a necessity for agentic workflows, and not a nice-to-have.

What’s Next for Scaling DataOps

I hope these lessons learned from building Petri can give you a jump-start in exploring how to build your own micro-orchestrator using agents. With the book finally done and having more free time to build again, I am excited to get back to writing long-form content on this newsletter. I have some awesome content planned as I dive further into AI agents and explore how to scale their infrastructure. Specifically, I decided to go deeper into this space and purchased an RTX Pro 6000 Blackwell GPU workstation to start building with fully open-source AI models and fine-tuning them for my specific use cases. Just like this article, as I build and learn, I’ll be sure to share my lessons with you all here!

Subscribe now

With the Snowflake and Databricks conferences finally over (and were thankfully on different weeks this year), the big news out of both is the open-sourcing of their data catalogs— Snowflake’s Polaris and DataBricks Unity Catalog. These moves track with my belief that we are moving away from building data catalogs for reference and instead building catalogs for automation based on the metadata. In my talk at Chill Data Summit a few months ago, I gave the analogy of this transition being similar to moving from physical maps to apps such as Google Maps. While data will always be “gold,” we are seeing the value of metadata grow tremendously and become a new battleground for vendors to control.

A huge thank you to this newsletter edition’s sponsor:

Commoncog: Teach Your Business Users To Become More Data-Driven

The hardest thing in data is teaching your stakeholders what you can do for them. Most data tool vendors teach technologies, not concepts. Here’s a free series that explains how to teach your business to become more data-driven. No tools, no buzzwords, no fads. Just simple ideas you can use tomorrow.

Hear from Jose Gerardo Pineda Galindo, Startup Advisor (Data):

Hear from "XYZ" highlights real-world use cases for all of us to learn best practices and upcoming trends within the DataOps space. I first met Jose via the Data Quality Camp Slack I managed last year, and we have collaborated on projects. In addition, he is very active on the Data Engineering Things Slack, providing mentorship to data engineers trying to grow in their careers. Not only is Jose a talented engineer and leader, but he is also someone who goes out of his way to share best practices and give back to the data community at large; thus, I’m beyond excited for you to hear from him and his lessons throughout his data career.

A previous startup you worked at describes itself as a white label Instacart for e-commerce brands, which I imagine requires a tremendous amount of integrations. What were the problems you faced when scaling all of these disparate data sources and how do you manage it today?

Jose: “Yeah, in this sense, we need to understand that it's a startup, so you get all the startup issues already, and that's the first thing that you get challenged on. And the other thing is because you are developing all the integrations, you get the problem of integrating existing systems, right? It’s a good way to have everything blank so you can choose how to do it. But then you get these customers who already have their systems, and because they already have their systems, you get the challenge of whether the system they chose is up to date. Sometimes, they have very old systems, and that's the first problem and biggest challenge because many things were already there that was really bad.

Then, the first thing I always do in this situation is map the process. The process is the first thing I need to understand because what are you trying to do? Why did you do it this way? What step-by-step process did you take to create this? So I can think of how to improve it, change it, or be realistic and say, okay, you know what, this cannot be optimized or improved. We need to build something from scratch. And that's one of the most important things: you never patch over bad processes. If you create an integration over a bad one or try to patch it, you get into huge problems.

So, for me, there is a process. We need to map it. We need to know the stakeholders. We need to understand the outcome. And then after that. We start working on the integration. So that's the multiple things that I face here, from integrating with external customers to integrating with manual processes or even external systems. So, I have faced these kinds of problems, and fortunately, we have managed most of them, but there are still others that are not possible to some extent. So I try to semi-automate it, and at least I can do that.”

In your last response, you mentioned “you never patch over bad processes.” Can you share a particular experience where you learned this lesson?

Jose: “There's this existing process that we had for financial services and systems, and I got asked to optimize it and improve it because it's taking a lot of time to load, but we need it tomorrow, right? That was the first hint. Okay, I know this will fail badly, but I need to do something now. There's nothing wrong with saying that I need to do it now; the problem comes later when you do that. What happened is that I created Let's say a semi-automated patch there to, pass through other systems that were there in place. And the problem there was that once it started to work, let's say a week after, then we started to get issues. You're probably not bringing all the data, or it crashed. Why did it crash? Oh, yes, I forgot these existing elements in this other process or these microservices.

And then, instead of building something better, I was spending all my time trying to fix these small or big issues; it was daily, “Oh, it's broken again. It's broken.” So, I tasked one of my engineers to keep track of it, and I built the process from scratch. Okay, it's going to take me a week and a half, maybe, but it's going to be worth it. So after that, I mapped what we were doing and what was expected to be done for the outcome. I talked to stakeholders, “Okay, how do you need the process to be? What is the most challenging part of your process? What if it's completely not needed?” So all of that took me about four or five days to map, create, architect, and design, and probably another week of actually developing it. After that, everything went smoothly.

It's something that I already knew, but you need to understand that for quick and dirty, sometimes you need to say no because it will take you more time later. Then, it will pile up with other technical debt, and at the end of the year, you will be looking back and saying, “Oh, I have a lot of things to do, right?” And that's what we need to avoid. There's no way patching something will be more worthwhile than building something new that will take you less time to maintain the whole system. That's a rule in my life for how I architect systems and how I develop things.

To elaborate a little bit on that and to finish this part, when I arrived at the startup—I hope my VP is not going to read this— he asked me, “Hey, you know what? Let's develop some pipelines on Elixir. You need to learn Elixir.” And I was like, yeah, I'm not going to do that. He built one pipeline. He showed it to me, and the performance was so bad because Elixir was not created for that; it was entirely different. So, a very important thing about this kind of ask is that you say “no” and push back completely, right? With arguments, of course, and my arguments were like, look, no one in the data engineering industry is using Elixir. I will never use it, probably maybe 20 years from now, but not now. It is used for other things. So many other things can do this 10 times better, so I'm not going to maintain your pipeline. And his pipeline was running at night and taking all the processes and all the performance out of our systems. And it took probably two to three hours to actually run.

Obviously, I said no; it took us about two weeks to design something very different, and there were no more problems since that pipeline was causing so many issues to so many areas. You have to stop and say, “No, think about the future.” And even if, when you are a startup, you need to pick your battles, right? And if that battle is going to become a Frankenstein a year from now, then either you keep doing that, but in parallel, you will need to be developing something to replace this old monster you're building.”

You have worked in retail, telecom, and network domains within data at varying levels. Across these domains what are unique data problems you faced that serve as key lessons in your career?

Jose: “I would say telco is the master there because telcos, in general, are like dinosaurs, and they develop systems themselves and everything. It's vendor-locked or was at least 10 years or eight years ago. Because of this, there was a very unique way of how data was converted and sent from one system to another. So, it was impossible to actually share data with another vendor that had a transformation tool or a database. It was horrible, I'm telling you, it's still horrible. That's probably why I moved to retail.

However, one of the biggest lessons I learned about how badly the telcos were doing in understanding their data-driven culture was with KPIs, which were very hard to develop. They're trying to standardize stuff, but at the end of the day, this standardization became even worse over time, I would say. With the arrival of 5G, they tried to change to this new IT world because what I can tell you is IT and telco have been separated probably for 20 years with differences in how they think and how they develop their systems.

And one of the things when they tried to change was the payloads on how 5G data was shared between one system and the other. And when I actually tried to read from this data, I realized how bad it was because they did seven, six, probably five, levels of nested JSONs. What are they thinking? And then the other thing was that they repeated the keys on the JSON. So, there was no way that you could do a normal query on those on the NoSQL database; there was no way. And to use it for other machine learning purposes, transforming that data into that payload into actually something useful was so horrible, I'm telling you. I probably spent a month and a half developing a system, extracting these data, deduplicating the keys, and extending the tree. I think that my skills in Python at that time went from advanced to super expert, I would believe because it was so difficult and very complicated.

At the end of the day, I think I did a very good job getting the best out of that data, but it's still a problem. It still happens because they don't have the right people to develop these systems. There are some other very good initiatives outside MACMA that are great; they're trying to get out of those protocols or old protocols that the telco industry developed. But I think that was one of the biggest masters for me, the telco domain with their horrible treatment of data there, the humongous amount of information, and, I would say, useless information.

I would say that 80 percent of the information is probably useless for data and machine learning. The things that you do on a daily basis when monitoring a system are useful, but for other things like anomaly detection, pattern recognition, monitoring, or observability, we just use 20 percent of that data. The problem is that the amount of data they dump is already huge. So I would say that would be one of the biggest things that even improve my skills at some level.”

You became a data leader, beyond the IC path, in the middle of the pandemic when work became remote. What advice can you give to other leaders on effectively leading data teams in a remote environment?

Jose: “So probably here, I had some advantage because I've been working remotely for probably 10 years already before the pandemic. So when the pandemic hit, it was like, okay, it's just the same for me, business as usual. But in general, as you said, working remotely and leading teams is not easy.

So, I do several things with my team. I like to be disruptive and very different, so I try to make it fun for them. If we have some refactoring to do or a new, you know, optimization to do for some scripts, I create a contest, for example, right? I gather them together and put them into teams. Put names to the teams, and then I say, okay, we're working on this, and it's very important, but we are giving a prize for whoever wins the contest; this kind of thing helps a lot on leading teams remotely makes it different for them. It's not just a person on the other side of the world where you're just chatting in Slack, and you don't even know the person. I tried to make them more collaborative in that sense so they don't feel isolated— not that engineers are super social, right? But at least bring them together through different activities. That's one of the key things.

The other is that you have to create your leaders and your organization, no matter how small or big. You need always to keep them on that career path. They will feel that they are also doing something valuable in their careers. Creating this small leader structure or giving some responsibilities to these people makes them also feel integrated in a very different way than just somewhere else typing and scripting things.

I would say that would be it. And I normally also do my monthly lean coffees. I don't know if you are related to lean coffee, but what I love about the lean coffee format is that it's not you talking as a leader. It’s you letting them decide what topic they want to talk about. So you generate this, okay, we're going to have our monthly review or monthly huddle if you want to call it, where we're going to put some topics and discoveries and new experiments things that they did this month that they think it's worth showing to the team.

Getting your team involved. It's very important. And the other thing is your one-on-ones. I think the one-on-ones are very important because they bring you together with them. I separate the one-on-ones that I have into technical and personal ones. So I try to make my one-on-ones not about the job because we have weekly meetings. We talk about that all the time. So when I have my one-on-ones, it's “Hey, what were you doing this weekend?” or “How's your family?” where you get to know each other. All of this really helps you to build a remote team where they don't feel that you are as remote as you would think.”

Person Profile:

Jose Gerardo Pineda Galindo is a startup advisor with experience in both implementing data engineering infrastructure and leading data teams. Feel free to connect with him on LinkedIn to learn more about his work.

What are others saying in the DataOps space?

"Good Enough" Data Models - Joe Reis

• What: A balanced take on the differences in how American and European companies approach data modeling and their implications.

• Why: Joe is writing a new book on data modeling and is someone who is really taking the time to understand the space—I will read anything about data modeling from him.

• Who: You are starting to explore data modeling and want to understand how it’s applied beyond theory.

Understanding Business Needs - Staying Relevant As A Data Team - Seattle Data Guy

• What: Insights on how to tie your data work to business needs and stay relevant.

• Why: The business is starting to become much more critical of the value data teams bring in relation to their high costs, especially with the rise of AI, so it’s in the data team’s best interest to ensure their work is aligned with the business's core needs.

• Who: You are a senior-level IC trying to identify which impactful data projects you should take on.

An Industry Shift: Moving From Collecting to Automating Metadata - Mark Freeman and Chad Sanderson

• What: Further elaborates on my above intro, giving an overview of Apache Iceberg’s underlying architecture and how the table format can be utilized to create self-healing data lakes.

• Why: Apache Iceberg is growing in popularity, and numerous major vendors are moving to integrate with it (e.g. Snowflake Polaris)

• Who: You are a data engineer trying to wrap their head around Apache Iceberg and the hype around it.

Subscribe now

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

Subscribe now

What is GTM Engineering

Hubspot defines a GTM strategy as “a step-by-step plan designed to bring a new product to market and drive demand. It helps identify a target audience, outline marketing and sales strategies, and align key stakeholders. While each product and market will be different, a well-crafted GTM strategy should identify a market problem and position the product as a solution.” While most startups start from scratch, Chad and I have a unique set of advantages:

1. We collectively have an audience of over 100K followers.

2. We have years of data tracking our content engagement and conversions.

3. Chad manually kept a spreadsheet of every meeting he had and who converted to design partners for the product.

4. We have the technical skills to analyze this data and build data products with it.

In other words, we don’t have an awareness problem. We have an optimization problem. How do we filter down and engage with the right potential customers when we expect to get millions of views of our messaging in a year? Even if a large portion of this audience are qualified leads, there is no way we can individually meet with all of them. We need to identify 1) who is the most engaged, 2) who is in our initial customer profile, and 3) whether it’s the right timing to engage with this lead. It became clear to us that traditional marketing methods would not work for us, and thus, we decided to pull from our engineering skills to solve this problem.

Thus, I define GTM Engineering as the process of using engineering problem-solving, data, and automation to bring a company and or product to market.

Problem Scoping

Even with the above advantages, it’s still challenging to determine what exactly a startup should focus on in its GTM efforts. A common mistake people make is jumping straight into tactics such as webinars, email campaigns, events, etc.. While this is great for quickly getting some traction for a startup, it doesn’t move the needle forward for market adoption. Thus, all of our GTM efforts at Gable start with the following five questions:

1. What does extraordinary success look like?

2. Who are similar people doing it, and how are they doing it?

3. How do we make this end goal possible?

4. What needs to exist for question “3” to be true?

5. Repeat questions “3” and “4” until you have a starting point.

What does extraordinary success look like?

For us, we are basing GTM success on the proxy of our ability to meet the milestones to raise a Series A funding round. For that to happen, we need to have X number of paying customers by the end of Y time. With your north star metric in place, you can begin to work backward.

Who are similar people doing it, and how are they doing it?

You don’t need to start from scratch, as you can leverage the power of hindsight to your advantage in your GTM strategy. I made a shortlist of founders who were in the data space and successfully launched their products, and I studied their online presence leading up to their company announcement. I’m talking going through every post, blog article, webinar, etc., to see how they built up excitement for their proposed solutions and got their first customers. Patterns start to emerge as to what’s effective or should be passed on for our unique business use case at Gable.

How do we make this end goal possible?

As stated earlier, Chad and I had a lot of data going into this GTM strategy development. For example, here is some of my personal LinkedIn data captured from Shield Analytics.

In addition, we have data on Substack engagement, Slack community engagement, Calendly meetings, and a list of design partners. With this data, we determine the conversion rates going from impressions on social media to securing design partners. With these conversions, we can work backward from our target customer count to each stage of the funnel to the number of impressions we need to reach.

Keep in mind that at this early stage, these numbers need to only be directional to help us establish a GTM strategy rooted in our unique business use case.

What needs to exist for question “3” to be true? Repeat until you have a starting point.

Now that we have our data, use cases, and targets, we can begin thinking through which tactics must exist to reach our end goal. The way Chad and I approached this was to:

1. Create categories for each major stage in our marketing funnel.

2. List out as many tactics as possible for each category.

3. Rank each tactic on a 1-10 scale for various attributes such as the reach of the tactic, how the tactic positions us in the market, or time commitment.

Based on these attributes, we can create a heuristic with a total score for each tactic. From there, we rank the best scores for each category and choose the top three tactics for each category. Finally, we ensure that these tactics will help us reach our proposed targets and iterate on the plan until we are confident that we can reach our targets.

GTM Architecture Design

If you thought building a data stack was difficult, try building a GTM stack, and you will quickly learn how disjointed this space is. Everything is held together by duct tape and Zapier integrations, and you need five demos with a sales executive before you even understand if the product is right for you. Even among seasoned sales and marketing professionals I interviewed for help, it was difficult to determine a stack for an early-stage startup that wasn’t cost-prohibitive. More importantly, most sales and marketing platforms are antiquated in that they rely heavily on email marketing. Thus, we had three primary criteria for determining a tool:

1. Ability to natively integrate with other GTM vendors without Zapier.

2. Access to an API for us to build our own integrations and automations.

3. Emphasis on social selling via LinkedIn.

Surprisingly, the resulting GTM tech stack closely aligns with a data stack, and we took full advantage of this. The below image is a drastically watered-down version of our GTM stack architecture, but you can see that similarly to any data workflow, our GTM captures marketing events, we log it into data storage, and then we leverage this data for insights and automation to drive revenue.

What makes viewing GTM stacks similar to a data stack powerful is that it becomes less about GTM tooling and, instead, about what architecture design aligns with with your business model.

GTM Optimization via Data Products

This is the fun part where I get to leverage my technical skills to drive our GTM strategy. Everything above was months of work to establish a strategy that would lead us to revenue and establish the foundational GTM architecture to achieve this. This is precisely what excites me the most about GTM engineering, as it ensures my technical work is directly tied to the business's revenue.

With the above architecture diagram of the flow of marketing data, actions, and expected conversions, we then identify where are the bottlenecks that we can automate. Thus, we create a product requirements document (PRD) to list every action in this value stream for GTM and then map those to technical requirements. Finally, we convert this PRD into Jira tickets and two-week sprints that align with the Engineering team at Gable (including doing sprint retrospectives).

While I can’t go into detail about what exactly I’m building for this data product, I can share that it’s a platform for:

1. Connect as many data points as possible in our marketing and sales efforts to our CRM HubSpot,

2. Identify new actions and or users and determine if they are in our initial customer profile (ICP).

3. If they are in our ICP, take specific actions such as a daily enriched leads list for the CEO or automated engagements.

The goal is to create the infrastructure to enable this automation to be easy to implement for semi-technical sales and marketing professionals.

My Vision for GTM Engineering

Through my data science roles, I quickly realized how little technical skills alone drove impact within a company– it was the combination of domain knowledge and the scalability of tech that was valuable. Thus, the primary purpose of GTM engineering is not to make more technical people involved with marketing and sales. Instead, the focus of GTM engineering is to make data and software engineering best practices as accessible as possible to slightly technical marketing and sales professionals.

Specifically, this looks like technical individuals building internal data products that 1) empower marketing and sales professionals to be data-driven, and 2) optimize their workflows to focus on the most important leads. More importantly, this requires making the technology as accessible as possible so that they can help develop these data products with their domain knowledge.

I’m currently building the V1 of our GTM Engine with Chad as my primary user to see if we can optimize his ability to drive people through a sales funnel and generate revenue. We are both confident in the potential of this platform, but things get interesting when we reach our series A funding round and start to build out the GTM and sales teams at Gable. If this resonates with you, then please reach out to me on LinkedIn as I am looking to build out a team to realize this GTM engineering vision when we begin to scale.

As I hinted before, earlier this year, I joined a stealth startup as the first employee focused on the data infrastructure space—I hope to share more details soon. Over the past month, I conducted interviews for a Head of Design role for our data infrastructure product. When we initially determined the attributes we wanted to hire for, I referred to this episode with Kai Hess, Founding Product Designer at Mage, as the first point of reference. This interview was one of my first, and few people have seen it yet, so I hope my new audience can gain valuable insights, just as I did last month while preparing for my interviews!

— Mark

What are your thoughts on design?

I was first formally introduced to design during my time at Stanford, with a huge emphasis on “design thinking” throughout the university. Every student I talked to said I had to get into one of the “design thinking for X” courses, especially if I’m considering entrepreneurship— you even had to apply to get into the classes because they were so popular.

Unfortunately, my grad school schedule didn’t allow me to take a course, but I did take multiple workshops to learn the design thinking process at a high level. In summary, the design thinking process consists of 5 steps:

1. Empathize

2. Define

3. Ideate

4. Prototype

5. Test

This process is actually how I approach many technical problems, with a major focus on empathizing. Even more interesting is how this theme of empathizing and defining problems is throughout my interview with Kai Hess below!

Hear from Kai Hess, Founding Product Designer at Mage:

Hear from "XYZ" highlights real-world use cases for all of us to learn from. Data collection sets the foundation for what's possible with our data workflows… yet we don't talk enough about it. A clear example of poor design hampering data processes is electronic health records. Doctors, the ones entering some of the most valuable data in the world, despise this product resulting in the messiest data one can work with. In contrast, it became clear to me how integral design is in capturing quality data when I had a chance to work with a designer on a data product feature. I'm excited to dive deeper into understanding the intersection of data and design with Kai Hess.

What is the role of design when creating data infrastructure products?

Kai: “The role of design is the same for basically any product that's created, and it is to provide utility and delight and function to the people that use it. Data scientists are a core part of the modern tech stack and modern business.

I mean, data is driving every single thing in Silicon Valley these days, and I want them to have a tool that is curated and designed specifically for them. And so anytime we're creating a product, whether it's internal or external facing, I'm trying to think about how do I make a intuitive experience that actually makes someone's day to day better?

And as this is a tool, that's what you're really focused on here. It's a data scientist that's going to be in something for hours a day, every day of the week, thinking hard about data, extracting data, working with charts. And so we need to make it so that it's the best possible and most intuitive experience.

And so the function of a designer more than anything else is to listen, and interpret, and understand the complaints and desires of the people that are using the actual tool. And so for a data focus tool, I'm talking with a lot of data scientists saying, "Hey, what are your pain points? What drives you crazy? What are you spending too much time on?" if you're spending time rewriting the same code every single day, there's a problem there or there's an opportunity there.

So design is really like being like a detective investigating where there are opportunities and problems in somebody's workflow. So for the tool that I'm working on right now, it's focused specifically on creating data pipelines for data engineers and data scientists.

We're figuring out how do we make that workflow better for them, better than it's ever been before, and design just gets to be a big push for making it right for those people.”

On your LinkedIn, you described one of your roles as “I made data sexy.” What's your process for making people pay more attention to data?

Kai: "So I am incredibly fortunate. That the people that I'm working with and the industry that I'm in, data is already paid way too much attention. I am working with data scientists and I'm working with tech people and BI people, and their whole world is data, so it's not really a matter of making that interesting to them or enticing to them cuz it's already those things.

The the role that I would have and the desire that I would have would be to make it effective. I think this is a problem with all data these days is that we are collecting so much stuff and really high quality stuff, but that's also disorganized. It's also got a lot of bad quality stuff hidden inside of it.

And you know, if you're making a snap decision, that's not a huge deal. But if you're driving business decisions, if you are training an AI model that's going to react and think about something. Your data has to be incredibly, incredibly high quality. And so a designer, their role would be to make sure that not only is data being moved effectively to the people that use it, but that they understand the efficacy of it, that they understand the transparency and the things that have happened to it before it gets to them. Because, that oversight, that understanding of everything that's happening to the data is the most important thing when it comes to having a quality assumptions based on something.”

How can data professionals best collaborate with designers to obtain high quality data within products?

Kai: “That's a good question. The better framing for that maybe would be, what can designers learn from data scientists and data engineers? I mentioned briefly earlier that designers are the conduit to intuitive experiences, delightful experiences, and effective experiences, but we are really just listeners.

We pay attention to what the people who are building the core functions and using the core features you're talking about. So data scientists, they should be giving the feedback on what they would like to see from design. We wish that we could effectively convey charts more. We wish that we could handle X, Y, Z type of scenarios that they're running into every day.

And a designer is kind of a liaison between all these groups where they get to look at both visual designs, interaction designs, even stuff like workflow designs and say, "Hey, let's see if we can make it better." So I think of myself as someone that is a facilitator just for progress within an organization.

And sometimes that just means sitting with a data scientist and seeing how they do things. I won't know necessarily what's going to be an opportunity or improvement, but just spending some time with them, getting to know kind of what their workflow is like is a really, really effective way to figure out how to design something progressive.

And I think one of the realities of work is that so often we're separated into silos. You know, product engineers hand off to product managers, data scientists hand off to somebody else. Research goes to these UX people and it trickles up, and everyone's kind of like talking, but you're in your own little unit. And so, for designers, it's really about breaking down those walls and seeing as a whole workflow, how do we create something that's great for the end user?

And that's what it's all about. Like we're not all engineering great things just for ourselves in private. We're making this for real people that are gonna use it. And so I don't always know because I'm a designer, I don't really know the depth of the problems that data scientists are dealing with. They are on a different level than me entirely when it comes to engineering and data innateness and their statistical ability.

It just blows me away. I kind of come in as a dumb little designer and I say, All right, what kind of stuff are you running into? Let me see if I can think of it with a totally different perspective. And so I think just creating that, that relationship between people is the biggest thing. Design's not gonna come in and push their agenda when it comes to design.

They're gonna listen and figure out what somebody needs and then go from there.”

Person Profile:

Kai Hess is the Founding Product Designer at Mage. I would highly encourage checking out his design work at https://www.mage.ai to see how he is helping data professionals (it’s stunning).

What are others saying in the DataOps space?

Council Post: Design Thinking And Data: The New Power Couple Of 2022

• What: A data executive describes five best practices in combining design thinking and data engineering.

• Why: This article provides a great reminder that our digital products with data are, at the end of the day, being utilized by real people.

• Who: You are a data engineer looking to implement more human-centric workflows with your data.

Data Quality by Design

• What: A thorough article on how to design for data quality in how products capture data.

• Why: Data capture is one of the most important steps in the data lifecycle and sets the foundation for what is possible with the data.

• Who: You are building a new product feature and want to ensure the data team is happy downstream.

Design Thinking Resources

• What: A repository of design thinking resources from IDEO.

• Why: IDEO has been one of the leading firms in the world for design since the 1990s.

• Who: You are brand new to design thinking and want to learn from some of the best.

  Subscribe now

The data market is changing drastically in 2023, and business fundamentals are more important than ever in a world where capital is no longer cheap. Though data initiatives have the potential to generate millions in revenue, many C-suite executives are beginning to question the viability of expensive data teams that are often burdened with high upfront costs and long-tail ROI. It’s not enough to have analytics and ship ML models– data teams must show their impact on the business’s strategy and bottom line. But how can data teams communicate this to leadership?

I recently came across a viral post on Twitter by Julia MacDonald that aligned with the processes I’ve used to successfully deliver high-impact data projects within various startups. Specifically, she shared her experience as a McKinsey consultant utilizing the Hypothesis-Driven Framework, in which leaders paid over $400k for their presentations. I highly encourage going through the original post on Twitter, but to quickly summarize the process, she detailed seven steps:

1. Gather the Facts

2. Formulate an Initial Hypothesis

3. Build an Issue Tree

4. Understand the Big Picture

5. Set the Stage with SPQA

6. Persuade with the Pyramid Principle

7. Make the Impact Clear

I had to learn more from her, so I reached out via DMs… and she responded. Our conversation resulted in the following case study that dives into how a data team could support leadership, via the Hypothesis-Driven Framework, after a major company mistake. Specifically, Epic System’s (electronic health record software company) recent blunder where their deployed ML classification model for sepsis performed worse than standard care within many hospitals.

Case Study — Epic’s Sepsis AI Blunder

Through my graduate studies at Stanford Medicine and work in the real-world evidence space in healthcare, one thing has become abundantly clear: the potential of AI in healthcare is massive… but the difficulty of safely deploying such models is even greater.

The Epic Sepsis Model is an excellent example of the challenges of such endeavors. According to a 2021 JAMA research paper:

The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.

This huge mistake resulted in Epic Systems pulling the product feature from hospitals, losing substantial money, and increasing scrutiny from regulators.

Using the Hypothesis-Driven Framework, how could a data team assist executives at Epic Systems in 1) recognizing the scope of impact, 2) the root cause of the issue, and 3) developing a path forward to reimplement the model to improve patient outcomes safely and recover a revenue stream?

1. Gather the Facts

According to the above 2021 JAMA research paper on the poor performance of the Epic Sepsis Model (ESM), the potential root cause of the failed model can be summed up into four points:

1. ESM missed 7% of Sepsis cases leading to late administration of antibiotics — this is worse than standard care.

2. ESM also caused “alert fatigue” via false positives for 18% of patients.

3. The algorithm behind ESM is proprietary and thus lacks peer-review performance documents, and there is minimal regulation for these models.

4. Epic is one of the largest electronic health record providers, leading to quick mass adoption in hospitals.

2. Formulate an Initial Hypothesis

From the above-gathered facts, a data team could generate the following hypothesis: 

“The Epic Sepsis Model is performing poorly as it is trained on national-level data but can’t be retrained for local-level differences within hospitals due to the model being proprietary and thus opaque.”

3. Build an Issue Tree

This massively complex problem can go in numerous directions for causes. Rather than get stuck in analysis paralysis, Julia recommended in her original post to break a large problem into multiple smaller problems that are much easier to manage.

Below is the issue tree that a data team could create detailing the potential problems of the Epic Sepsis Model:

4. Understand the Big Picture

Now the data team can take the individual issues and craft a story of the big-picture problem being experienced by Epic and provide a recommendation:

“Despite Epic having one of the largest electronic health record datasets available, the Epic Sepsis Model model built on this data is generalized on a national level but not to a specific hospital level.

Coupled with Epic being a proprietary software with one of the largest market shares, this model was adopted quickly without being validated in the respective hospitals it was deployed in.

This led to 7% of false negatives and 18% of false positives, resulting in below-standard care that did not align with initial study results released by Epic — leading to further scrutiny from regulators, Epic pulling the product feature, and lost trust among healthcare providers.

We recommend that the Epic Sepsis Model be retrained on data specific to a respective hospital and only if the hospital meets the inclusion criteria to retrain the model successfully.”

5. Set the Stage with SPQA

SPQA stands for situation, problem, question, and answer. Something data professionals, including myself, struggle with is going way too deep into details. This is one of the quickest ways to lose the interest of executive leadership. Thus, SPQA is an excellent tool to quickly provide them with the needed information. If an effective data team were providing details to Epic’s leadership, I would imagine they would share the following:

Situation:

Epic released a product feature utilizing AI to classify if users are at risk for sepsis. This resulted in below standard of care for patients, the product feature being pulled, and regulator scrutiny.

Problem(s):

The released proprietary model was trained on national-level data but performed poorly when faced with local-level data within hospitals.

Coupled with a large market share, the propriety model was quickly adopted through Epic’s distribution channels without hospitals being able to validate for their respective populations under the false security of national-level results.

Question:

Can Epic release the sepsis model again while addressing the poor performance at a local hospital level?

Answer:

Yes, by making the product feature less opaque and allowing hospitals to retrain the sepsis model on data that are representative of their respective hospital population.

6. Persuade with the Pyramid Principle

Describing the problem to executive leadership is not enough for them to take action; a data team needs to persuade leadership that the issue and solution they are providing must be prioritized now. The Pyramid Principle is highly effective at doing such.

7. Make the Impact Clear

Now that the data team has determined the root cause and has buy-in from executive leadership that the problem needs to be addressed, the data team has to sell a vision of how they can alleviate the pain of the business. This is one of the most crucial steps in illustrating the data team’s impact on the business’s strategy and bottom line. I can imagine the data team suggesting the following to Epic’s executive team:

1. Work with regulators to fully understand the problem and repair trust with the medical community.

2. Identify CTOs of major hospital systems using Epic EHR software to build partnerships with key influencers of the healthcare market to retry the Epic Sepsis Model.

3. Build a software platform that allows hospital data teams to securely access and retrain the sepsis model with data representative of their respective hospital.

4. Create stronger inclusion and exclusion criteria for hospitals eligible to use the sepsis model in which they have the infrastructure to retrain the model to their specific hospital.

5. Utilize a neutral third party, such as a university healthcare system not using Epic, to validate the new model while accounting for the heterogeneity of different hospital locations.

Real-World Result

Though the above is just a thought exercise, it was rooted in real-life events that impacted patients' lives. In October 2022, Stat News released an article with the following:

… Epic is now recommending that its model be trained on a hospital’s own data before clinical use, a major shift aimed at ensuring its predictions are relevant to the actual patient population a hospital treats.

Conclusion

I hope this case study on utilizing the Hypothesis-Driven Framework can help you drive more value in your organizations and highlight to leadership how your data initiatives impact the business’s strategy and bottom line. Again, I want to thank Julia MacDonald for allowing me to leverage her original Twitter thread and providing feedback on this blog post. I highly encourage following her on LinkedIn and Twitter to see more of her helpful content.

Referenced Sources:

1. Commissioner, O. of the. (n.d.). Real-world evidence. U.S. Food and Drug Administration. Retrieved May 5, 2023, from https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence#:~:text=What%20is%20RWE%3F,derived%20from%20analysis%20of%20RWD.

2. Habib, A. R., Lin, A. L., & Grant, R. W. (2021). The epic sepsis model falls short—the importance of external validation. JAMA Internal Medicine, 181(8), 1040. https://doi.org/10.1001/jamainternmed.2021.3333

3. MacDonald, J. (2022, December 16). At McKinsey, we charged $400K+ per presentation.here's the simple 7-step framework we used (steal it for free)🧵: Twitter. Retrieved May 5, 2023, from https://twitter.com/julia_m_mac/status/1603727229053173767

4. Ranadive, A. (2013, June 21). The pyramid principle. Medium. Retrieved May 5, 2023, from https://medium.com/lessons-from-mckinsey/the-pyramid-principle-f0885dd3c5c7

5. Ross, C. (2022, September 30). Epic overhauls popular sepsis algorithm criticized for faulty alarms. STAT. Retrieved May 5, 2023, from https://www.statnews.com/2022/10/03/epic-sepsis-algorithm-revamp-training/

6. Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine. https://doi.org/10.1001/jamainternmed.2021.2626

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

A lot has changed since this episode was originally released in October 2022. Specifically, Ben has been promoted from VP to CTO of Magenta Care Continuum, where he s continuing his push for high-quality healthcare data! If you want to hear his perspective from the CTO angle, then I highly encourage checking out this panel I did with him as well:

Scaling DataOps Newsletter

SDO 018 - What Happens When Your Infrastructure Doesn’t Scale Anymore? - Panel Discussion

Hear from… an entire panel of experts: This edition of Scaling DataOps is a little different in that I have three technical leaders sharing their insights. In January, we did a live version of the Scaling DataOps Newsletter at the Data Teams Summit, which was awesome. I’ve taken the highlights of this panel from each speaker to share with you all! At the…

Read more

3 years ago · 3 likes · On the Mark Data

— Mark

What are your thoughts on healthcare data?

Why is healthcare so slow in adopting AI and precision medicine within the US? There are a multitude of reasons— policy, high regulation, etc.— but the largest challenge is data quality. In my first data science role, I worked with a registry dataset with 80% of all ophthalmology records in the US, and it was one of the hardest datasets I had ever navigated. This startup had an exceptional data model and an exceptional data engineering team… but we didn’t control the source data. Since US healthcare is so fractured, every hospital (even different departments) has different electronic health record software (e.g. EPIC). Even within the same software, they would have different configurations for collecting data. Scale that up to any data product, and you are going to be slowed down by a myriad of data quality issues. If we want to advance data in healthcare, we have to drastically improve data quality, and it’s why I’m so obsessed with DataOps.

Subscribe now

Hear from Ben Doremus, VP Technical Operations at Magenta Care Continuum:

Hear from "XYZ" highlights real-world use cases for all of us to learn from. I often joke with my friends that you must be a masochist to enjoy working with healthcare data due to the messiness and complexity. It has everything: high regulation, high stakes of improving or harming lives, and high domain knowledge, all collected by electronic health records that are universally hated by doctors entering the data. One person who has repeatedly jumped into the chaos of health data to bring order and insights is Ben Doremus.

Electronic healthcare data is some of the messiest data I have worked with. What are your top lessons from scaling data pipelines in this domain?

Ben: “I mean, it just comes down to trust no one and trust no one data source either. Like you gotta have multiple sources of truth. It's one thing that we found time and time again. When we get data, we get data from revenue cycle, we get it from clinical, we get it from quality. We try to get data from different systems that don't talk to each other so that we can try from there to triangulate to the truth for what's really going on.

Because especially when you're looking for like clinical truth, healthcare data is based on claims, It's based on billing cycles. It's not actually centered on any of what's really going on clinically. You can try to extrapolate, you can use, you know, SNOMED ontologies or whatever else you like, but it's still all based on billing codes, so you never really know what's going on.

So, multiple sources of truth. If you can get the notes from the physicians, that's a gold mine, but they also use templates. So it's super messy there as well, so you can't trust it. We found multiple notes where within the same note, the provider refers to the person as male and female inside the same note.

So like you just using a template, this is not helping anyone here. So yeah, multiple sources of truth, that is like a hundred percent where you start. But then once you've got your multiple sources of truth, you're still kind of in hot water. What do you do when you have conflicts? What do you do when you find logical inconsistencies in the data?

And this is where I would say 80 to 90% of my work has come, is in figuring out how to reconcile those situations. So it's really all coming down to. Business rules and business logic, and for anybody who's implemented reams of business logic, before you know that it's really hard to get all of that to align, to be logically consistent.

It's really hard to keep it up to date to keep these things consistent. If you go from ICD 9 to ICD 10, what are you gonna do? How are you gonna deal with that? So all of this business logic implementation has to be done really carefully in a way that's nice and modular, easy to keep up to date, well documented.

I hear people go on about how, ‘Hey, I don't need to comment my code because it is self-referencing.’ It doesn't work when you need context. That doesn't work when you've got business implementations. You gotta have really good contextual comments to make all that stay up to date and usable a year or two down the road when all of your assumptions fall down.

You have to have access to clinicians and coders. They're not the same thing. They have totally different skill sets, clinicians and coders, and you need both, and they need to be like at your elbow while you're doing this. One of the great things that we did at the first healthcare company I was at.

Was we actually mixed up our teams. So there wasn't a tech side and there wasn't a clinical side. They were blended teams. So the team that I oversaw had nurses on it. They were part of the team. Their responsibility was the same as the software engineers. To make sure that the data we're putting out makes clinical sense and getting those people working together is really hard.

So you gotta have the right people to do it. You gotta people who want to learn, who are curious, who want to expand their horizons, both on software side and the clinical side, right? Like you've got a clinical person who doesn't want to learn how to fix their computer, they're not gonna do great in this scenario.

But there are so many nurses from the ORs and so many coders who want to expand their horizons and learn more and grow and figure out how is this thing called SQL works? Like if you can empower them and get these people together. Oh, it's magic. It's so cool, like a different version of multiple sources of truth, right?

It's like do you have the coders defining the truth and the clinicians defining the truth and the software engineers defining the truth. You need everyone together and like that's how you work with really messy data is you never trust it and you get multiple opinions and you figure out how to make sure those opinions are scalable and maintainable.”

Something that's important but not discussed enough is data security. How do you ensure you keep patient data protected as you scale these complex data systems?

Ben: “So having been at startups, you get to wear multiple hats, and I've had the pleasure of going through three different HITRUST implementations now, and the first time I rolled my eyes the whole time, I'm like, ‘You're gonna make me do what?’ It's, you know, some of these things are written a decade ago, and they're all about in person work and dial up connections and things like that where you're like, ‘Oh, this is ridiculous.’

And the second time through I was like, ‘Oh. I see why they did this now. This is what they're protecting with that. And I can ignore all the other stuff.’ And now like I'm right in the middle of my third time, and this is the first time I've really done it from scratch, where I've got a blank slate, a company that hasn't existed yet, and I get to literally build the company with compliance in mind.

And it's been a game changer for me to view it that way. So, If I could suggest to anyone what to do, it'd be build the company with compliance in mind. Have a framework identified early. HITRUST is actually a really good one. I've been really impressed with it. We've gotten some template policies and procedures that tell us, ‘here's all the controls you need to hit,’ and once you can step back and really see the whole landscape of what you're trying to protect with this, it makes the nuance of each individual control way, way easier to implement and understand contextually. When you just get this big all list of, ‘here's my security checklist.’ It sucks. That is drudgery at that point, right? But if you can start to build a system out of it rather than just little check marks, it really works like it really comes together in a nice way.

And you know, there's some new technologies that make this really easy too. Identity management providers you know, Okta's and, and stuff like that. Holy cow. This just makes it so much easier to manage your user access and your permission levels, and it makes it easier to gather all these things and, you know, stuff didn't exist just a few years ago when I was doing this, or at least it wasn't popular.

So it's, there's, there's more and more things coming on that aren't necessarily, built for healthcare IT and all of that. But really, if you know the general landscape of what's available, it's way easier now than it used to be. Especially with cloud providers, right? Like they take a huge burden off when you're using the cloud rather than having all these on-prem things.

I know it's a totally different story when you're a bigger company and a totally different story if you are a healthcare provider yourself. But from my seat of small companies aiming to help healthcare be better, it's not as bad as it used to be. It's getting even better, and there's all these new laws coming out around interoperability that I am itching to see if they make it dent the problem because they also help with security. You know, they themselves have their own controls in place and they take off some of the burden that we have to right now in sending and receiving data securely. So it's ever changing. The, the landscape is always on a move and staying on top of what's available to you is probably the most important thing.

Have a framework and know what your options are I'd say, boom, there's your two.”

Not only have you scaled data systems, you have scaled data teams as well. What guides you in growing high performant data teams tackling the complexity of healthcare?

Ben: “This is really hard to answer because even in the five years I've been doing this, I have seen so much variety in what is considered a data team. I've had teams that I would term production analytics, we call them data engineers, and I've had teams where we were serving essentially a data infrastructure as a self-service platform, data engineers.

I've had teams that were really just converting JSON to CSV, data engineers. Like all of these different things fall under the same title. So when you ask for, ‘give me advice on building data teams,’ well, like what type? They're all so different. The job I had prior to this one, I was in charge of two different data engineering teams working on two different products with two widely different skill sets.

There was zero overlap between them, like seriously, zero overlap. And it was, the context switching for me was brutal for it, right? Because on one team I'd be like, ‘all right, we are implementing a domain specific dbt model to solve this problem, and the other side we're trying to deal with airflow scaling issues.’

There's just no technical consistency between them, but there are still a lot of things you can do in terms of managing any team at scale. Some of the things I mentioned in the first question. Pull together the content experts with the people doing the coding. You can't separate them.

And to that point, some of the worst performers were people who had blinders on and said, ‘Not my job.’ People who said, ‘I was told to do this, so I did this,’ and they didn't gather context, and they didn't go figure out why they were being asked to do it. That's a recipe for failure. Probably in any job anywhere, but especially in the data space where context is so important.

So curiosity, I guess what it is, if you can foster curiosity and empower people to have a broader ownership over what they're doing, rather than just, ‘I got my job, I do my thing, here's my ticket, check.’ That is just necessary. You can't scale a team of any size. You can't start a team of any size if they don't have that curiosity, but it definitely won't work at scale because one of the problems that I had earlier too was like we had disparate people coming into the data team.

I came in from data science, these other people came in from analytics, and people had their projects that came with them. So then they said, ‘This is my vertical. These are the projects that I take. I take this side of the ticket and then I'm done.’ And when we moved from the tech team and services team to the blended model, another thing we did was we said,’ tearing down all the walls, you don't have projects anymore, that is not your code base, this is everyone's code base.’ You have to be able to move in and out of all of the different projects so that you can see the whole ecosystem of what we're dealing with. Otherwise, you're just gonna make assumptions that are bad. So now you gotta have a primary code owner.

They can handle all the PRs and all that, but they shouldn't be doing all the coding. You need other people in there. You can't have silos is really what it is. You can't have silos in technical skills. You can't have silos in the code that's being worked on. You do want the code to be separated though, especially early on.

Modularity is super, super important because things are gonna thrive and things are gonna die. So building a modular code base, very important. But don't let people find their niche and hold on to it and say, ‘This part is mine and I don't need to know anything else.’"

Person Profile:

Ben Doremus is the Vice President of Technical Operations at Magenta Care Continuum. Feel free to connect with him on LinkedIn to learn more about his work.

What are others saying in the DataOps space?

Data Pipelines in the Healthcare Industry

• What: A great intro describing the clinical workflows and how it’s represented in data pipelines.

• Why: Healthcare requires so much domain knowledge to start, this article can help.

• Who: You are relatively new to healthcare and what to gain a high level overview of working with such data.

Artificial Intelligence and Machine Learning in Software

• What: The U.S. FDA finally released its guidelines on using AI and ML within healthcare after years of anticipation.

• Why: This is a BIG DEAL, as companies finally have guidelines on implementing AI and ML software within clinical workflows and whether or not it’s considered a medical device.

• Who: You are creating production ML systems within clinical settings.

How to build an effective DataOps team

• What: High-level overview of the various roles within an effective DataOps team.

• Why: Insights into what roles are needed, how they support DataOps efforts, and how to tailor it to your organization.

• Who: You are a leader planning data strategy and or headcount.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

What are your thoughts on being a “data-driven organization?”

Upon completing this newsletter edition, I can only think of one thing: I just stumbled across a hidden diamond in the data landscape that more people need to pay attention to. The type of organization that my guest, Jhakir Miah, described was nothing short of a dream company with respect to data maturity. For context, Amrock is within the Rocket Companies (previously branded as Quicken Loans) portfolio, and their work reflects the culture of the broader Rocket Companies ecosystem. Specifically, their relentless pursuit of automation and the lockstep between data teams and business leaders is next level— and they are one of the few companies that are truly “data-driven” throughout the entire portfolio. Below is a great example of what we aspire to accomplish with data!

— Mark

Hear from Jhakir Miah, Director of Engineering at Amrock:

LinkedIn never ceases to amaze me with its potential to connect you with amazing professionals in the data space. One of those professionals is Jhakir Miah, who popped up in my post feed with amazing insights on data leadership. Though this interview was our first conversation, I hope this will not be our last, as Jhakir has a wealth of insights on technical and business leadership. I learned a lot in our brief chat, and I’m excited for you to learn about him and his team’s amazing work.

Sponsorship:

Real quick… my goal is to keep this newsletter free for my audience and have sponsors pay. Engaging with the following would be a huge help if you want to support my newsletter!

This edition of Scaling DataOps is sponsored by SingleStore who is hosting the free webinar: Build a ChatGPT App on Your Own Data - April 25th, 10 AM PST.

You can register for the event using my featured link, where every signup goes a long way in supporting the Scaling DataOps Newsletter:

LINK TO REGISTER

In addition to supply chains, Covid lockdown completely changed the real estate market essentially overnight. How do such drastic changes impact how your team works with housing data?

Jhakir: “So I work for a company that has multiple different smaller sister companies within the portfolio. Rocket Companies as a whole has been in the mortgage industry and this housing market for over 35 years. So these changes that we saw in the market. They threw off a lot of these younger companies and startup companies because they have not went through the cyclical cycle and the shifts in the market.

Rocket companies been through this for so many years that when the shift happened, we immediately knew how to redirect our businesses. When Covid hit, we knew that, okay, refinance is gonna start booming because the interest rate was going down. So immediately, we had the trained professionals and the staff, and we started engaging more people to start focusing on the refinance side and captured that part of the market while maintaining our purchase market as well.

It never impacted us, it didn't hinder us at that point because, like I said, we've relied heavily on technology as well. So we immediately started leveraging data. Where in the market do we need to go to, which areas do we need to focus on to really optimize our processes? Which business process can we automate using data that makes sense to say, "Okay, this can be done through computers, while these should be done with humans."

Because they've been through this so many times, our leaders like to say that this is like a baseball game. This is the fifth inning or sixth inning. So if we're down, that's okay. We still got six or seven more innings left to play. So at that point, they knew what to expect. They've been through this several times. They shifted our focus accordingly and our strategy adjusted to ensure that we are able to still sustain what we had, as well as capturing the new market that was coming around. When that Covid pipeline happened, everyone was home, but the housing market, saw a huge and significant rise in the needs and the demands, and we just need to figure out how to support those business needs.”

Mark: “So, just to follow up on that. How do you align your data strategy with that? Because there's such a historical precedent already there, but data changes so quick.”

Jhakir: “Yes, absolutely. You said historically, organization has always been data-driven. So it was always engaged with our business stakeholders and partners. Before they even made the decisions, what does the data say. Then we validated that against the market side. Does that make sense? And our senior leaders and our business partners with our experience, they said, "Okay, it does make sense, we will anticipate this." So the data teams automatically knew where to go because the business, we were along with it, and we were on the same table discussing those changes.

Before final decisions were made, data teams came in and validated a lot of those things. We said, "Okay, is this the right approach for us? Does that make sense? What does it say?" And the company is very tech-driven. Even though on the mortgage industry, we service housing, much of our sister companies do the Rocket Auto or Rocket Loans, we have Rocket Money in our portfolio as well right now, but they're tech-driven. So a lot of times they'll say, "yes, this makes sense. We think this is a good market for us to go into. Let's validate that. What does the data say?" They bring in the data people to answer those and validate those, then make decisions accordingly.”

One of the largest players to be impacted by AI models changing during the pandemic was Zillow Offers losing over $500MM due to a poor AI model. What considerations do you make to ensure your data processes don’t meet a similar fate?

Jhakir: “That was a big one. That was a wake up call for a lot of organizations really leveraging AI and data science models to say, "This is the direction we go." For ourselves, ensuring the data quality aspect of it is there. We're validating these things at a very smaller scale first to test them to validate or confirm those before we scale it out to most. And because of that industry experience over the years, so even if before we make a decision to purchase $500 million, it is validated against industry expert is validating against senior leadership who's been through the cycle to understand "yeah, sure we're not hesitant to make large investments where it makes sense. But we're thoughtful in that, does the data confirm what we believe? Is there evidence to support that? We should be able to explain what the model's doing."

So a lot of time it's on the data scientist to explain what the models are doing, how it's doing, why it's to be there. We have positions where we've seen senior executives in our business partners in a meeting with the data scientist or a data analyst, walking through each piece. It doesn't trickle up for us, where we don't say, "Oh, this analyst did it, so the VP's gonna go present it." No, bring the analyst to the table, let's have a discussion, because it's that person doing the analysis, doing the data checks, and validating everything. Let's talk it out. You're the expert in the table, you have the center seat, let's figure out if this is the right decision we as a company need to go.

I saw an article that resonated really well with me. Instead of it being data-driven, it should be decision-driven analytics. So here's the decision we wanna make. We'll find out everything that needs to be there. Find the data, validate it, and build your model. See if it supports the decision. The other way around is it's gonna be a tremendous amount of shifts that needs to happen, mindset and growth. The business becomes the reactive component as opposed to us, the data people being the proactive and saying, "Hey, this is the decision you wanna make? Here's how we proactively make it. Here are the data sets that support that or deny that."

A lot of times we've been in conversations where business partners want to make a decision about going one direction. We've looked up the dataset. You can go, but it doesn't make sense. You're not gonna make as much money as you think you're going to make because here's what the data says historically, here's how the data says if we were to go pre-covid how we did. So being able to have that type of voice in our organization is really impactful. Then you can say, "Okay, we are data drifting because when we're wrong, we're equipped to admit that we're wrong. Okay, we'll try something else. But when we're right, we go all in on it."

Mark: “Wow. And I imagine it probably takes a substantial amount of trust to be built over the years to have that. Because many times you'll have leaders who are like, "Well make the data work to align with my decision." Or I'm very anchored on this kind of sunk cost fallacy; even though you're giving me this data, I'm still moving ahead.”

Jhakir: “Absolutely. And that's where like I started with Rocket Companies, which used to be Quicken Loans, Rocket Mortgage, all of those sort of rebranding happened and we became part of a larger whole. But I've been with Rocket Companies for about three years now in the mortgage industry. So I've been in the culture that it embeds in this is that we are gonna automate what we need to automate because we want our people to focus on the most important and the critical piece. So if a data can support that 100%, go behind it. And when I started three years ago, I was a little taking back of how well the data teams are, for lack of a better word, the prestige that the data team has, right? Generally, it's always like, "all right, we'll make a decision and we'll look at the report after the fact, right?"

It's always, we have a retroactive look at and that's when data people comes in, but even now I thought, okay, wow, this is great. We made even more strides since I joined. Where now data people are in the forefront of the decision makings. We're the ones that, any migration that happens, any technology upgrade, what does the data people say? We need to make sure the data peoples are on the table so that we don't lose insights, we make sure we gain, as opposed to losing anything when these migrations happen.”

What advice can you give other data leaders to help them better navigate the data challenges caused by black swan events?

Jhakir: “So I think if you've been in the data space long enough, we can think of black swan events as once in a lifetime never happen and things of that nature. But if you've been in data space long enough, you are preconditioned to deal with things breaking out of the norm out of nowhere, right? It's the nature of what we work is just unbelievable how the smallest little thing can have the biggest effect and it trickle down everywhere.

So we're very much used to these type of black swan event, even though we may not categorize them as black swan events, we're very used to these type of reactive approaches that we have to take in order to ensure sustainability, reliability, and all of those components of our infrastructure. And it happens to us very more frequently than we'd like to admit. Our things break.

So the biggest advice is in twofold. Go deeper into your tech stacks technology, and the technology am I say go deeper into it means when you're building a robust system, there are things you taking into consideration. Failures will happen, things will break. Will it scale? Do you have the resources you need to train up and down your team members, right? Do you have the resources to expand your environment? Infrastructure as needed? So go deeper into that and make sure that is sustainable. And when we build the system that is designed to do that, you already account for any type of black swans or production failures, whatever you wanna call them.

The second component is building that trust with your business partners. I've been fortunate enough to work with business partners where if I come to 'em and say, "Hey, listen, something broke. I can't deliver this today. They said, okay, don't worry about it. Can we get a next week?" If you don't have that relationship with them, they lose that trust, they lose that need to come to you. If they lose the need to come to data people, then you become disconnected from your people who are actually doing the work at driving the business, and that's when conflicts start to happen. That's when you really start to question whether you are a data-driven organization. So building that relationship and having a transparent communication with your business partner is gonna be the second component that's gonna save you no matter what those event happen.

Because as an organization, black swan doesn't happen just to you as data leader. It's gonna happen to the entire organization. And if you have those relationship, those strong, meaningful relationship, and you have that trust with them, you can say, "Listen, we're all in this together. I just have to happen to oversee the data components. You oversee sales or marketing, we're all gonna be impacted. How do we work together so that we as an organization can move forward?"

So, own your stuff. Make sure it's built to be resilient enough and that goes into, like you said, we already do that as a tech people. Kneel into it and make sure you truly believe in that, that it can scale, that when chaos happens it won't break and it will go as supported and have the trust and communication with your business partner to be able to communicate, for lack of a better word, when shit hits the fan. That's when you really need to come down and have a kumbaya and say, "okay, how do I make sure we are still sustaining and growing and moving to the future?”

Person Profile:

Jhakir Miah is the Director of Engineering at Amrock. Feel free to connect with him on LinkedIn to learn more about his work.

What are others saying in the DataOps space?

Evolution of Data Platform at GoDaddy

• What: Technical blog that covers the journey towards building a modern, low-cost cloud data platform that prioritizes scalability, reliability, cost-effectiveness, security, and governance. It includes the early days of data at GoDaddy and the best practices for establishing a well-defined data strategy.

• Why: This blog provides valuable insights and guidance for organizations embarking on a similar journey to build a successful cloud data platform. By sharing the lessons learned and experiences from the journey at GoDaddy, readers can gain a deeper understanding of the key considerations and best practices for building a successful cloud data platform.

• Who: This blog is a must-read for those who want to gain a deeper understanding of the key considerations and best practices for building a successful cloud data platform and learn from the lessons and experiences of GoDaddy.

The $500mm+ Debacle at Zillow Offers – What Went Wrong with the AI Models?

• What: Zillow, an online real estate marketplace, closed down Zillow Offers due to inaccurate property valuations resulting in a $500 million reduction in Q3 and Q4's estimated value.

• Why: Zillow's algorithms overestimated home values and didn't adjust when the housing market cooled down. The issue was caused by "concept drift" in machine learning models, which assume the past equals the future, and did not account for rapidly shifting values or market shocks.

• Who: Technical leaders in the data industry should consider leveraging better tools to monitor and maintain AI models' quality, including measuring model accuracy, outputs, and inputs to detect potential model issues.

Data Mesh And Strategy Tech Stack Alignment

• What: A discussion on the challenges businesses face when implementing macro technology solutions, particularly in the data platform space.

• Why: The article provides insight into the need for best-in-class ecosystems and the importance of capturing business context when building data platforms.

• Who: You are responsible for implementing data solutions for the business and are interested in learning about the challenges associated with building technology platforms such as data mesh.

What are your thoughts on data governance?

Getting the business to care about data infrastructure can be an uphill battle, given how abstract data is. Combined with the underpinnings of a data product or dashboard being hidden in plain sight, it’s hard for non-data people to feel the need for data infrastructure. Yet one of the strongest levers to get the business to pay attention is through data governance.

Gartner defines data governance as “…the specification of decision rights and an accountability framework to ensure the appropriate behavior in the valuation, creation, consumption and control of data and analytics.” In other words, reduce the risks associated with data and thus the risk of losing revenue. This is especially apparent in regulated industries, like finance or healthcare, where data leaks can cost companies millions; such as Anthem being fined $16M for a data breach. Even non-regulated industries face risk without proper data governance, including Bird who “overstated its revenue for more than two years by recognizing unpaid customer rides.”

These risks only become amplified as your company scales to the enterprise level. This is exactly why I’m so excited for you to hear from Tiankai Feng about his experience in data governance within a 60K+ employee organization.

— Mark

Hear from Tiankai Feng, Data Governance Leader & Musician:

Tiankai Feng is an extremely talented data professional, has worked at one of the top brands in the world, and worked on some extremely interesting data projects at the enterprise scale… but that isn’t what caught my attention. What caught my attention was Tiankai’s creative music combining data, singing, and the piano! Prior to my passion for data was my love of dancing, where I used to street perform and was part of a dance crew. Thus, I always gravitate to creatives such as Tiankai, as that creativity allows you to view the world in a unique way. The moment I heard one of his songs, I knew I had to interview him on this newsletter to learn how he thinks about data.

Your most recent position was heavily focused on data governance within product. What makes this role so difficult, and how can data teams best navigate the complexities of data governance?

Tiankai: “Very good question. I think data governance generally depends heavily on the data domain that you're in, because with different data types and different data domains, you have different challenges. Being in the product data domain means, especially in companies like Adidas or any, let's say footwear or apparel company, that it's really the core of what a company does. Basically, how revenue is generated. That means also that in these organizations, many people are working with real data. So it's a very complicated stakeholder landscape.

Also, I would say that as compared to more regulated industries like the financial industry or the pharmaceutical industry, the footwear and apparel industry is-- besides some sustainability regulations-- not as much regulated, and that makes pushing data governance and data quality measures a lot harder because it's all based on good intent and convincing people, and it's not externally pushed basically to do so.

And lastly, I would say that there's really a lot of different knowledge and motivation levels across different stakeholders. So balancing the resistance versus the advocacy of it all and making it all work together is really hard as well.

So in total, I would say in summary, that means for me in the triangle of let's say, people, process, and technologies, the people part is definitely the hardest, and it also requires the most focus when it comes to data governance. So I'm usually paying a lot of attention to communication and relationship building as the key pillars for successful data governance.”

What are the unique data challenges experienced by enterprises, as large as Adidas, and how do you navigate them as a leader?

Tiankai: “I would generally say in data management, you have the data creator side and the data consumer side, right? And especially when you have a big company, that means you have a high amount of data creators and a high amount of data consumers. And the bigger the organization is, the less they talk to each other rather simply put.

And that means there's just a bigger symmetry of information and knowledge and all of the misunderstandings too. That means that there's a lot of mismatch regarding the intended purpose of data and the actual use case of data, and then that often leads then to dissatisfaction and sometimes even conflicting requirements towards data quality.

If you put on top then the ever-changing IT landscape of different data systems being replaced or changed or upgraded or whatever, that makes it even an operational challenge because as a data governance team, you would have to balance projects where you have basically helping with like new systems being integrated, versus the actual end-to-end governance on an existing landscape.

This, having said, is what I think are the biggest challenges in a big enterprise. And the only, I think, direction to make it better is really open transparency and being honest with each other on what is going on. And if you have a central place, like a data catalog where you can already centralize a lot of the transparency there, that would be nice. But you'll still need that communication element to let everybody proactively tell each other that things are happening and there are certain data being created and certain data being used to really avoid these misunderstandings. I think a big part of it also is assessing with stakeholders together about the impact of data, because you also want to make sure you prioritize the right things. So it all goes back again to just transparency and communication, I would say.”

You are one of the most musically inclined data leaders I know. What made you decide to mix music with data, and how do you integrate music into your leadership style?

Tiankai: “Yeah, it's a great question. I don't think I get that question often enough, to be honest. I'm very glad to answer that. So maybe just for context, I started playing piano when I was five; admittedly forced by my father because it was his dream and he put it on me because he couldn't learn it when he was little. But I was only liking playing piano when I was 10. And that journey from being forced to and actually seeing the value itself in playing the piano already gave me a lot of starting points on what happened and how I would bring it to others to do things they might not enjoy in the beginning. But either way, I started writing songs when I was 15. I played at a bar pianist during my study years as well, which was a really good gig that I had.

So music has always been a big part of my life. I just hadn't shown that side actually too much on the professional channels yet before, that only happened now a few years ago. And I think for me, actually, music and data have more common than people think because the analogy would be that music consists of certain notes, right? And notes are predefined in the frequency physically and everything. It's just like data points that are predefined with certain values, right? But how you put it all together, how you visualize it, how you tell a story around it, or how you basically phrase it. This is when artists create it. So basically you combine it all, use different variations of it, and that makes it art from a very technical point of view, from very scientific point of view, all of a sudden becomes art.

And this is how I would see working with data as well. It doesn't have to be all super scientific if you can make art out of it. And there's so much flexibility now in the new world to actually make art out of this. So in this way, my music mindset actually I think makes me a better leader because I try to inject creativity in everything I do and also would work with that with my team members on it.

And I think for those people that don't work in data, that actually they have the opposite perception. They think it's like this numbers crunching, Excel opening job, that everybody just typing numbers all the time, but it really isn't anymore. And at this point, data people have gotten so good as well in communicating that it's just a new world of working on data.

Lastly, I would say making music together is a whole thing on its own right? Being in a band or being like a jazz big band makes you individually perform as part of a bigger thing. But you have to listen very actively to everybody else too, to actually fit in. And that is for me, like the ne plus ultra definition of teamwork, right? Where it's not only about you performing, but knowing what everybody else wants and where they're at, and basically being part of the bigger whole.

And this is something that I think also gave me a lot about just meeting culture or collaboration aspects to be not only the talker but also the listener. And then basically driving it together as teamwork. So yeah, I think that is how I would basically see how my music self is influencing my data leader self. And I'm very happy I can combine the two now, actually a lot more.”

Person Profile:

Tiankai Feng is a data governance leader and musician. Feel free to connect with him on LinkedIn to learn more about his work, as well as catch his latest data songs. His exact words were "I’m always looking for exchanging knowledge and best practices [with people].”

What are others saying in the DataOps space?

HBR - What’s Your Data Strategy? The key is to balance offense and defense.

• What: “In this article we describe a new framework for building a robust data strategy that can be applied across industries and levels of data maturity.”

• Why: “Data was once critical to only a few back-office processes, such as payroll and accounting. Today it is central to any business, and the importance of managing it strategically is only growing.”

• Who: You are a leader within your organization tasked with establishing a data strategy.

MIT Technology Review Insights - Report: Modern data architectures fuel innovation

• What: A 2023 report from MIT sharing trends on data infrastructure and its impact on the business.

• Why: “Inadequate data management has substantial costs, slowing companies’ access to strategic insights and impeding the implementation of advanced technology such as AI.”

• Who: You are someone tasked with developing the roadmap for your company’s data infrastructure.

Is bigger always better? A controversial journey to the center of machine learning design, with uses and misuses of big data for predicting water meter failures

• What: A 2019 research paper describing how the data quality of their training data led to poor results, and their methodology for addressing this issue.

• Why: This is a great real-world example of Andrew Ng’s call for “data-centric AI.”

• Who: You are someone interested in how you can improve data quality for your ML models.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

I just spent the past week at Data Council Austin, learning from some amazing data professionals and, more importantly, getting a snapshot of the data landscape. Among the great talks, one thing became very clear: we have a lot of data startups with substantial overlap, whether it be data catalogs, data observability, or now the emerging generative AI space. Is this fractured market sustainable with a looming recession and capital becoming expensive again?

This week I’m holding off on the interviews and diving deeper into the above question by synthesizing my observations of the market, conversations with previous guests, and the various conversations I had the past week in Austin. In this edition, I aim to answer the following:

1. How did we get to our current market in the data landscape?

2. What’s the outlook for data startups today?

3. What does the changing market mean for data professionals?

While I can’t predict the future, I hope to give you the market context so you can better navigate the changing landscape as it unfolds.

Upcoming SDO Interviews:

• Black Swan Events in Housing Data - Jhakir Miah

• Navigating Enterprise-Scale Data Challenges - Tiankai Feng

• How to Upskill in DataOps - Sarah Floris

How did we get to our current market in the data landscape?

The past fifteen years have not been normal between the 2008 housing crisis and the coronavirus pandemic— both of which put us on the path of surplus that eventually has to end. Specifically, I argue that our current data market resulted from:

1. The creation of cheap capital via monetary policies enacted by the US government to curb crises.

2. The rise of the cloud which drastically sped up startups’ time to market for their solutions.

3. Snowflake’s IPO pushing investors heavily into data, with its peak in 2021.

With interest rates rising again, the underlying assumptions of the heavy investments into data no longer hold. Thus, the data industry is on a collision course with a harsh reality.

Though the 2008 crash feels like a completely different lifetime, its reverberations are still felt today in the form of extremely low-interest rates that made capital cheap. From 2008 to 2022, we had near-zero interest rates, which completely changed how capital was viewed by financial institutions and, thus, venture-backed companies. This is most apparent in the rise in valuations for seed stage and series A companies and their ability to raise subsequent rounds with minimal revenue.

Fast forward to March 2020, and it looked like the decade plus bull run would finally end as the entire globe ground to a halt. What proceeded was some of the largest gains in technology we have ever seen, with subsequently even more money being pumped into the data startup ecosystem by venture capital.

While one could state that the shift (shove?) to remote work increased the value of technology, I would argue that monetary policies once again bolstered the US economy to curb a financial crisis in the form of stimulus checks and PPE loans for businesses; ultimately kicking the can further down the road.

So with monetary policies making both capital cheap and bolstering the economy during crises, we understand how investors had the means to invest in data companies, but why did they decide to invest in data? In addition to the above “black swan events,” two market shifts specific to data were 1) the rise of cloud infrastructure and 2) the Snowflake IPO in 2020.

The cloud as we know it was first introduced by AWS in 2006 and didn’t reach a $100BN plus market size until 2016. In that time, we had an explosion of SaaS startups as the barrier of entry was diminished to a few button clicks and a credit card payment. The competitive advantage for companies was now speed to market, as getting your product in front of users and iterating would give you the “escape velocity” venture capital drooled over. This created the perfect environment for the emergence and rapid adoption of the Modern Data Stack and the various point solutions supporting the data lifecycle. Also, there was that HBR article resulting in every company rushing to hire a data team regardless of need.

These data companies fit perfectly within the venture capital model as well. Every company in the world has to use data; therefore, data companies have the $1BN plus total addressable market to perk up the ears of VCs. In addition, with the growing open-source and cloud ecosystem, the means to build these companies had relatively low initial costs compared to traditional hardware companies. But what accelerated investment into data companies to the peak we saw in 2021 was the 2020 Snowflake IPO. Below is an excerpt from my interview with Matt Turck, Managing Director at FirstMark Capital, explaining the situation:

In the wake of the Snowflake IPO, there was as we all know, an enormous amount of excitement around data infrastructure, the rise of the Modern Data Stack, all those things.

So you end up with a bunch of very interesting companies getting started and a lot of venture capital that was more than happy to fund those companies and then fund them again, and then six months later, fund them again. Occasionally again and again. And that was, a lot of fun, a little dizzying, but ultimately led to a lot of categories emerging overnight and getting very crowded overnight.

The Snowflake IPO “validated” (heavy emphasis on quotes) for venture capital the market opportunity in data infrastructure companies, leading to the race to get their own piece in the space. The below tweet by Matt Turck best illustrates this in seeing the landscape in 2012 and now 2023.

All of which brings me to the present day, where I stood in the vendor room at Data Council Austin and saw numerous data startups essentially doing the same thing. Many of these companies raised unbelievable valuations in the past few years in a fractured market where consumers have a swath of options for every step in the data lifecycle. Before, revenue didn’t matter as companies could be propped up by venture capital as they sold a vision of their market opportunity. Now, consumers are slashing their budgets in a down market, and financial institutions have less capital to deploy with rising interest rates (i.e., interest rates finally moving back to normal levels). In short, our world of surplus is over, the underlying assumptions behind these investments are no longer true, and these data startups with huge valuations have a rough seas ahead of them.

What’s the outlook of data startups today?

The 2023 bank run on SVB was the wake-up call to data startups that their previous world of surplus was now over. The below tweet gives a great summary of the SVB situation, but in summary, the increasing interest rates by the Feds led to SVB’s deep investment in long-term bonds to become realized losses. This spooked VC investors, and a prisoner’s dilemma was presented to every founder who deposited more than $250k in SVB. Though the federal government thankfully stepped in to make depositors whole and avoid the collapse of the regional banking system in the US, a strong signal was sent to our market: what happens when startups run out of capital?

My previous interview with Ethan Aaron, Founder & CEO at Portable, perfectly described the position many companies now find themselves in the wake of SVB’s collapse:

… [Let's] rewind back to 2021 and 2022. Interest rates were very low, which means valuations of everything were very… high. You could have one dollar revenue and the value of your company was a thousand dollars. We saw the same thing happening in the data world. Companies with very little revenue seen unbelievably high valuations…

What does that mean? It means that those companies raised a lot of money. So if one of these companies raised a hundred million dollars… you don't really have to worry about these companies disappearing overnight. That's not the problem that is going to face companies that took on money at too high of a valuation.

The problem these data startups face now is growing into these unbelievably high valuations in a saturated market where businesses are slashing budgets and head count. There is not enough available revenue in the market for all of these data companies to be healthy businesses. In addition, they can no longer buy time to reach product-market-fit via an infusion from venture capital without accepting a down round (if they can even raise again).

More importantly, Ethan also highlighted, in another one of our conversations off-mic, how these companies’ high valuations but low revenue make it unlikely for them to exit via acquisition. Instead, I argue that we will be left with highly valued zombie companies that can’t be acquired and limp along without product-market-fit for years to come. Some may use their long runway to pivot, but ultimately they are chasing growth rather than creating it.

What does the changing market mean for data professionals?

I often hear people say, “we will finally have a much-needed consolidation of data companies,” but I disagree. As I stated earlier, we will have zombie companies limping along with their massive bank accounts and valuations but with minimal customers to cater to. I argue that it’s in data professionals’ best interest to understand the risk profile of the vendors they currently utilize and are considering. Some key questions to ask yourself when evaluating this risk:

• Is the vendor solving a business problem tied to revenue or just providing a point solution for a specific area of the data lifecycle?

• When was the respective vendor’s last round, and how much money did they raise?

• Given their valuation, do you believe their total addressable market warrants such a valuation?

• Has the solution provided by the vendor reached product-market-fit or is it on a solid path to such a state?

• If a respective vendor were to go out of business or pivot, how difficult would it be to migrate to a new solution?

Even if you are not a vendor, it would be naive to think this massive market shift won’t change the relationship of data teams with the business. The time of POCs going nowhere, R&D without a strategy, and raising cloud costs is now over. We had a great run on the data hype cycle, but now we need to actually deliver the outsized value we promised or risk the same fate as vendors of being cut from budgets.

How does one deliver outsized value with data? You become a strategic partner to the business to determine how to mitigate risk in a changing market or generate revenue. Data professionals are in a unique position where we have asymmetrical access to information about the business that many in the org are not privy to. The quicker you can utilize this advantage to impact a business's bottom line, the more at ease you can feel about data’s role in your respective business. If you need examples of achieving this, has two great articles I highly recommend on repositioning your data career in the wake of potential layoffs:

High ROI Data Science

Developing And Evolving A Data Organizational Structure To Meet Business Needs

I have built data organizations for the last 8 years. Before that, I built and led teams in traditional software engineering organizations. Organizational design and development require repetition to master. Most companies struggle because they don’t have the experience…

Read more

4 years ago · 6 likes · Vin Vashishta

High ROI Data Science

What To Watch Next In The Layoff Cycle

In late 2021, laggard companies like Peloton were the leading indicators of deeper trouble to come. Businesses that are already in trouble are the first to respond to changing market conditions. This…

Read more

4 years ago · 4 likes · Vin Vashishta

Closing Thoughts

Though I highlighted the role of venture capital in propping up our data industry, I don’t believe it’s fair to blame them for creating an unsustainable market in data. I hope this newsletter edition illustrates how their actions are mainly a symptom of unique market conditions. Specifically, historically low-interest rates were utilized to curb financial crises in the US, as well as the evolution of technology shifting us to the cloud and data, which changed the dynamics of startups.

With that said, the symptoms still exist, and the data industry has a rising fever. Eventually, something has to give due to rising interest rates breaking all investment assumptions. Though the ideal solution is seeing a bundling of all these overlapping data vendors, this is just not feasible given their massive valuations and limited avenues to reach revenue to match, thus limiting acquisitions. We have a turbulent road ahead of us in the data industry, but I remain hopeful. The swing back to actually providing value and generating revenue means that our relatively young industry is being pushed to mature, which is better for all of us in the long run.

Referenced Sources:

1. Data scientist: The sexiest job of the 21st Century. Harvard Business Review. (2022, October 19). Retrieved April 1, 2023, from https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century

2. FDIC acts to protect all depositors of the former Silicon Valley Bank, Santa Clara, California. FDIC. (n.d.). Retrieved April 1, 2023, from https://www.fdic.gov/news/press-releases/2023/pr23019.html

3. The Federal Reserve. (n.d.). Implementation note issued March 22, 2023. Board of Governors of the Federal Reserve System. Retrieved April 1, 2023, from https://www.federalreserve.gov/newsevents/pressreleases/monetary20230322a1.htm#:~:text=The%20Board%20of%20Governors%20of,%2C%20effective%20March%2023%2C%202023.

4. Freeman, M. (2023, March 10). SDO 017 - navigating the ML, AI, and data (MAD) landscape as a VC - Matt Turck. Scaling DataOps Newsletter. Retrieved April 1, 2023, from https://scalingdataops.substack.com/p/sdo-017-navigating-the-ml-ai-and

5. Freeman, M. (2023, March 4). SDO 016 - navigating uncertain markets as a data leader - Ethan Aaron. Scaling DataOps Newsletter. Retrieved April 1, 2023, from https://scalingdataops.substack.com/p/sdo-016-navigating-uncertain-markets

6. Kaji, S. (2023, March 9). A lot of panic re: SVB (you should see my phone/emails!). A bank run driven by panic is the real risk here, not the action of selling LT Securities at Lossi have no inside information as I left SVB in 2012, but know enough about banking to piece together. quick 🧵. Twitter. Retrieved April 1, 2023, from https://twitter[.]com/Samirkaji/status/1633958266509336576?s=20

7. Lee, J. L. (2020, July 8). Data shows companies that raised funds in 2020 also approved for U.S. PPP Loans. Reuters. Retrieved April 1, 2023, from https://www.reuters.com/article/us-health-coronavirus-ppp-funding/data-shows-companies-that-raised-funds-in-2020-also-approved-for-u-s-ppp-loans-idUSKBN2490EW

8. Snowflake announces pricing of Initial Public Offering. Snowflake. (2021, July 7). Retrieved April 1, 2023, from https://www.snowflake.com/news/snowflake-announces-pricing-of-initial-public-offering/

9. Temkin, M. (2022, June 7). The market correction has come for series A and seed startups. PitchBook. Retrieved April 1, 2023, from https://pitchbook.com/news/articles/Series-A-seed-deals-venture-capital-market-turmoil

10. Turck, M. (2023, February 23). How it how it's started going (2012) (2023) pic.twitter.com/lce4rekxqb. Twitter. Retrieved April 1, 2023, from https://twitter[.]com/mattturck/status/1628879218535763968?s=20

11. Vailshery, L. S. (2022, December 6). Public cloud computing market worldwide 2008-2020. Statista. Retrieved April 1, 2023, from https://www.statista.com/statistics/510350/worldwide-public-cloud-computing/

12. Vashishta, V. (2022, December 4). Developing and evolving a data organizational structure to meet business needs. High ROI Data Science. Retrieved April 1, 2023, from https://vinvashishta.substack.com/p/developing-and-evolving-a-data-organizational

13. Vashishta, V. (2023, January 25). What to watch next in the layoff cycle. High ROI Data Science. Retrieved April 1, 2023, from https://vinvashishta.substack.com/p/what-to-watch-next-in-the-layoff

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

Hear from… an entire panel of experts:

This edition of Scaling DataOps is a little different in that I have three technical leaders sharing their insights. In January, we did a live version of the Scaling DataOps Newsletter at the Data Teams Summit, which was awesome. I’ve taken the highlights of this panel from each speaker to share with you all! At the time of this recording, we had the following guests with respective titles [follow them on LinkedIn]:

• Sarah Floris - Senior Data & ML Engineer, and founder of Dutch Engineering

• Ben Doremus: Chief Technology Officer - Magenta Care Continuum

• Richad Nieves-Becker: Sr. Associate VP, Data Science - Revantage

My goal with this panel was to look at a complex problem from three different perspectives of experienced IC, senior manager, and c-suite… and my data friends DELIVERED on this panel. I highly encourage you to check out the full panel, as there are some awesome discussions that I didn’t have space to include here. Enjoy!

— Markl

Real quick… my goal is to keep this newsletter free for my audience and have sponsors pay. Engaging with the following would be a huge help if you want to support this newsletter!

This edition of the Scaling DataOps Newsletter is sponsored by SingleStore who is hosting the free webinar: How to Build GPT-chatbot Apps with SingleStore & MindsDB on March 30th at 10 AM PST.

You can register for the event using my featured link, where every signup goes a long way in supporting the Scaling DataOps newsletter:

LINK TO REGISTER

What are the early signs that your data infrastructure isn't scaling anymore? Specifically, what is the impact you see on the technology, product roadmap, and data teams themselves?

Sarah: “This a problem that I see on a regular basis, right? You're scaling your small startup, you're medium company, you're scaling and suddenly you get all of these warnings from places. And that's hopefully only if you have monitoring.

Otherwise, your pipeline just start crashing, your data is stale, the things that you see on a regular basis when your infrastructure doesn't scale anymore. And so, when I see it as a data platform engineer, I am immediately alerted to that, and then I have to go in and do those reactive changes, and it becomes a very much more reactive environment, which is not always the best way to approach these problems, right?

So the first thing I always typically want to do when I start a new data project or things like that: have monitoring. Usually, the first sign is getting those monitors, having those alerts set up, right? And so that's really what happens. You start having a lot of people, customers, stakeholders, leadership come to you and be like, "Hey, what is happening to my data?" And so that often happens. So the first one is the backup of logs. That's what you all see in the data teams, you'll get a lot of folks who can't take on any more requests because we are being so reactive in our responses to the data pipelines.

And then the other thing is the projects are not getting done. You'll see the technology also giving warning signs, you'll see leaders starting to complain, but you'll also see customers come in and talk about what is happening.

And that is usually like when you have an embedded analytics product. And so those are the things that I would definitely watch out for What are your data engineers doing? What are your data scientists doing? Making sure data analysts, what are their complaints? And a lot of the time you'll hear a common theme and that's, like the queries are taking slower than they're supposed to be. And then, the data engineers are not getting my work. I'm not receiving those of the data for those ad hoc requests. And so those are definitely something you have to really watch out for.”

Scaling often puts you within the build vs. buy debate to meet your new data needs. As a leader, how do you navigate through this decision while also accounting for the needs of your growing data team?

Ben: “This is a very easy question and a very hard one at the same time. If you've got a wide open budget, you've got the options of doing whatever you want. If you have constraints, then you gotta figure out what's the most valuable. And I think this really comes down to a philosophy of who you are as a company.

You look at this with like the cloud, right? Why is the cloud so popular? Because people don't wanna be managing their own infrastructure. That's not the strength of their business. They don't want that taking up their brain space. They wanna focus on solving the problems and just have computers exist so that they can use them.

Yeah, it's more expensive, but sometimes that's worth it. You know, it's not all just a cost thing, it's who are you, what is the thing that you wanna focus on? Can you clear out your brain by offloading all these other things? You know, everyone's got Salesforce. Nobody wants to build the integrations. That's not a part of any company is managing their own Salesforce integrations.

It's just buy that, get that figured out somewhere else. But there's also really interesting questions between like, okay, do you buy it from a bespoke company. Do you get a consulting firm to build this for you? Do you go with an open source thing and try to hack it on your top? A lot of that still comes back to your philosophy as a company.

Who is it we want to be as a team? Do we want to scale up in this? Sometimes it's worth it to say like, "it's gonna take us longer, it's probably not gonna save us a ton of money, but people wanna do this, they want to give it a shot." I'm not talking resume driven development here, but there's something to be said for using the strengths in the way that people want to use them, allowing them to grow and allowing them to stretch.

So, in a world where everything's free, you just offload everything except what you need to focus on as a business. But when it's not, you just need to figure out what is the truest to who we are? What can we afford and what can we get away with for now? Not buying until we're really gonna need it later.”

Investment to scale data infrastructure is unique because it requires high upfront costs for long-term ROI. How do you get buy-in from leadership to make such investments when the pain of data scaling isn't felt yet by the broader organization?

Richad: “There's a couple options. Nothing speaks like the pain of experience, right? So I think the best way to do this is to simply give an executive a slow app. Infrastructure sells itself. Here's all these logical arguments, or you could just give them the slow app, that you built MVP style, which you should, no matter how big your company is. Right? And then you're like "okay. Is this valuable? Oh yeah. Yes. By the way, it's slow. In order to make it not slow, we need X, Y, Z." So I would say that's one way to do it.

The more broad answer is storytelling. So instead of giving them that experience. You can also paint a picture of what the experience will be like in a very concrete manner. Because ultimately I think things that you can't see always come back to the things that you can, the outcomes, the dashboards, the ML, the performance.

And so basically you have to often yourself think and do the creative work, hard work of figuring out how it ties to business outcomes. You also can tie it into where you want to be in three years. If you are defining a roadmap or you have one defined, you could point to those things and say, "this will be very difficult to do with this many customers versus these few customers," or "this is gonna be really difficult when we scale to Europe and Asia versus just here." So you basically use things that you know, they already care about as hooks and then you hang your idea on them.

I think Ben gave a good example where it's like, We hire the team ourselves and then you draw out the cost for them. Or you buy this thing and maybe you try to point out or estimate, okay, where does buying the thing become more expensive? And you say " by the time we get there, we'll be concerned about these other things." You try to make it fairly airtight, but it's impossible to make it fully airtight. Ultimately buy-in is people's beliefs, and the beliefs are changed one person at a time, one experience, and one story at a time.

General principles, tips, I guess work backwards from value. Show don't tell, and hopefully you're working with a team with a long-term view because if, back to the first question, if everyone's super reactive, it's gonna be very difficult. There's no magic answer to convince anyone of anything. If we had that, then there would be peace in the world and stuff, right? But obviously you can't always convince everyone that of everything you need. Ultimately it does depend a little bit on your environment. There's no perfect answer.”

What are others saying in the DataOps space?

Meaningful metrics: How data sharpened the focus of product teams

• What: Duolingo highlights how moving away from aggregate data for key product metrics opened up new insights and possibilities.

• Why: This is a great article to learn more about the challenges faced by our data colleagues towards the end of data pipelines.

• Who: You either work in or support a product-focused team with your data infrastructure.

Timeless Obstacle for Data Products: Data Quality

• What: A great overview of the various components of data quality within a product.

• Why: The author highlights how ubiquitous data is in all products and how data quality is essential to a product’s success.

• Who: You are starting to take data quality seriously within your org and want a starting point to understand how.

Scalable Annotation Service — Marken

• What: Data labeling is essential for ML and search within a data product and Netflix shares how they do this at scale with Marken.

• Why: The article includes an awesome architecture diagram and in-depth explanations.

• Who: You love seeing the innovations of big tech players and looking into how you can potentially apply them yourself.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

I recently joined a stealth data startup as employee one (I will share more details soon), and my life has been consumed with thinking about go-to-market, ideal customer profiles, sales funnels, and everything in between. It’s simultaneously exhilarating and exhausting joining a startup this early, but I’m legit living my dream through this role.

Related to my current life, I wanted to share one of the earlier newsletter episodes (SDO 002) few have seen regarding sales. This interview was in October 2022, when I was going through rounds of vendor calls searching for data infra at my previous job. A few of those calls highlighted how sales teams could be an asset to your data team when they genuinely care for your pains and want to create a viable solution. Hence my conversation with Mike Ebbers on how data teams can effectively utilize sales to support your data initiatives. Enjoy!

— Mark

What are your thoughts on sales?

I will never forget my first day picking up the phone to call potential customers pitching the AI solution my co-founders, and I were building— in short, it was very humbling. That day gave me tremendous respect for my Sales colleagues’ tenacity and insight into how they could make me a better data professional. Specifically, Sales professionals are typically the first contact points with customers our data solutions will impact. Thus, by staying in contact with Sales, I can listen to the pain points that are surfaced and use that information to determine how to position my data projects to best provide value.

Hear from Mike Ebbers, Account Executive at Monte Carlo:

Hear from "XYZ" highlights real-world use cases for all of us to learn best practices and upcoming trends within the DataOps space. Sales may not be your first thought of being a data role, but they are uniquely positioned to see the challenges data teams are facing across the industry through their discovery calls. I talked with Mike to learn more about what he is seeing in the market for data observability.

What pain points are you finding data teams are struggling with the most in your discovery calls?

Mike: “The biggest one that almost all pain points boil down to is data trust. The primary problem comes from the following symptoms: data issues are not caught before downstream consumers, whether it be analysts, data scientists, or actual paying customers of a data analytics product.

They're the ones slacking the data engineering or DataOps teams going, "hey, this data doesn't look right," or "we know this is wrong," or "this value can't be true," or whatever the case may be erodes the trust between the organizations. That issue continues to be pervasive in other ways because then people don't use the data as much. If they don't trust it, they're not gonna use it, right? So you can't be data driven if you don't have trustworthy, reliable data.

The second one is I think, just data engineering teams more specifically, tend to live in a more reactive mode. They're not actually doing the things they were hired to do, like building pipelines and do data modeling and some of the stuff that actually provides value to the business. They're being brought into these reactive kind of firefighting situations where they have to figure out why there was a null rate in a column where there shouldn't have been or why the data didn't show up from that third party data source.

So it really comes down to those two. It's the data trust and then also being reactive.”

Data practitioners are the user of data solutions, but they often need to “sell up” to their leadership to make purchases. How have you seen data teams successfully “sell up” in this situation?

Mike: “The first thing is awareness of the problem. People don't provide solutions, and they don't fund solutions, where they don't understand what the problem is. And the corollary to that is what's the impact to that problem? And it's gotta go beyond, ‘Hey, my job is being filled with a bunch of annoyances,’ it actually has to have some impact on the business. So that's the first thing. Awareness at the executive level of the actual problem that exists.

The next thing is what's the quantifiable impact? Right? And actually there's two buckets to this. There's hard dollars, ‘are we actually losing money on a product, for example?’ And then there's also soft costs, which is, ‘we hired 15 data engineers and 50% of their time on a weekly basis is being spent on various issues.’

So it takes both sides, it's soft and hard costs. But a key aspect that data buyers, particularly data technical folks, don't always pick up on is understanding how your company makes these decisions. Because, if I'm in a bank I'm probably gonna talk dollars and cents with a greater emphasis than I would if I'm an up and coming startup who's still trying to build things out culturally. So you have to understand how your leaders actually make decisions.

Last thing I'll mention is consensus. What about your peers? Who else does this impact? If it's data engineering, or data science, or finance, or some other department, some domain specifically that has this problem-- does that cascade to other departments as well? Is it cascading from other departments? Let's understand that a bit more and build consensus around what we need to go do. And, the answer is not the same for two companies. You have to assess that and kind of understand what people have a tolerance for versus not.

So those would be my answers, right? That's how, that's how people successfully kind of sell a direction, whichever they choose.”

Sales, at its best, helps companies solve their pressing problems. How can data teams best leverage individuals from Sales in solving problems that warrant an external vendor?

Mike: “The first part is coming prepared with ‘here's what the problem is and here's our general perspective on why we need to go solve it.’ Good sales people are really good at understanding from a big picture perspective how this all fits together: the business, the technical aspects of it, all the different resources that are necessary for assessing whether or not there's a fit, supporting them from a product perspective, from a business perspective, creating the white papers or business cases.

But sales people can't do that unless they understand what that problem is they're actually trying to solve and how it connects to the business. And I would argue that if there is no connection to the business, it might not be worth solving that problem.

So you have to come with that hypothesis and perspective, then just communicate openly. The worst thing is to be working towards two different things for two different reasons. Being as aligned and streamlined as possible towards that mutual goal of solving the problem is critical, and that's how you have a healthy relationship.”

Person Profile:

Mike Ebbers is an Account Executive at Monte Carlo, where he ensures data teams are the first to know about and solve data issues via data observability. Please connect with him on LinkedIn to learn how he can help you.

What are others saying in the DataOps space?

Thoughts On Go To Market For Modern Software Infrastructure

• What: Perspectives from a Venture Capitalist on go-to-market approaches for software infrastructure.

• Why: The data infrastructure space is crowded, thus, you need to be intentional about standing out in the market.

• Who: You are bringing a new data product to market and are considering different approaches.

Use This 6-Step Approach to Get Buy-In for Data and Analytics Strategies

• What: Six tactics to help you sell internally to data leaders.

• Why: The C-Suite reads Gartner, so learn how to speak to their needs.

• Who: You are trying to build buy-in data infrastructure needs.

The Mindset of a Data Leader

• What: Tips derived from talking to 50+ data professionals and leaders on how to sell a data product.

• Why: It’s not enough to build a data product, you also have to make the value proposition clear to your potential customer.

• Who: You are a founder looking to obtain your startup’s first twenty customers.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

What are your thoughts on venture capital?

Last week we discussed the changing data market from a founder's perspective. For this edition, we are going to the other side of startups to understand the VC perspective. I highly encourage checking out the articles at the bottom of last week’s interview to get the historical context of our market. Then check out the MAD Landscape articles in this edition to understand the current context.

My first interaction with venture capital was in grad school when my co-founder and I applied to Pear VC for our health data startup idea. I quickly realized I was out of my depth as I was asked about TAM, competitors, and other details about the health market... rejected. We then applied to Lean Launchpad, got an interview, and again crumbled when VCs started questioning our business model… rejected. A few more rejections that year and a pattern became clear— VCs are some of the best individuals at identifying holes in your proposed ideas, for they are in the business of saying “no.”

The more I learned about venture capital, the more I understood why they must be quick to say “no.” VCs are in the challenging position of seeing hundreds of startup pitches but only having the ability to allocate funds to a handful of companies. In addition, venture capital is beholden to the “power law,” where a successful return for their respective firm’s fund is often driven by a handful of companies in their portfolio. It’s a tough job, but this is precisely why VCs are some of the best individuals to learn from concerning changing trends within markets.

— Mark

Hear from Matt Turck, Managing Director at FirstMark Capital:

Hear from "XYZ" highlights real-world use cases for all of us to learn best practices and upcoming trends within the DataOps space. One of the coolest things about creating content is that it can be a platform to enable you to meet people you look up to. Matt Turck is one such individual who has significantly shaped how I view the data market. Ever since I started my career in data, I have looked forward to Matt and his team’s release of the MAD Landscape and seeing how the data industry has grown. This landscape has inspired many of the questions I’ve asked leaders in this newsletter regarding how they are navigating such a fractured data market. So I’m beyond excited to share with you all this interview with Matt and learn where the data industry is potentially going in the future. Enjoy!

With the aggressively increased fed interest rates, institutional capital has become expensive again, thus directly impacting venture capital. What impact will this economic shift have on the MAD Landscape in the next few years?

Matt: “I do think that we are in a very different world all of a sudden. It's true for any kind of startup, but certainly for MAD including data infrastructure companies. In the wake of the Snowflake IPO, there was as we all know, an enormous amount of excitement around data infrastructure, the rise of the Modern Data Stack, all those things.

So you end up with a bunch of very interesting companies getting started and a lot of venture capital that was more than happy to fund those companies and then fund them again, and then six months later, fund them again. Occasionally, again and again. And that was a lot of fun and a little dizzying, but ultimately led to a lot of categories emerging overnight and getting very crowded overnight. And it was not a completely irrational approach because certainly the Snowflake IPO was an unlock for an entire space.

And if you believe, as I do that ultimately every company is a data company, meaning every company needs to have a data infrastructure and be data-driven, then the market for this is very large because the market is ultimately everyone, right?

So it was not completely irrational. At the same time, this led to a market that was super crowded, and now the music has stopped, and everyone needs to or is in the process of adapting to a new reality. And look, I'm a huge fan of the data ecosystem, machine learning, AI, and all the things. But equally, I think we should be all very transparent and honest with ourselves about what's happening, and what's happening is not necessarily a lot of fun. I think something's gonna have to give, right? You can't have a lot of very young, often single-feature kind of companies, many companies being below five million in ARR.

And then on the other hand you have customers who are probably gonna be much more selective and discerning in their buying process because they are gonna be under strict scrutiny by their CFO. A lot of supply, arguably at least for now, less demand. And then, on top of that, less easily available venture capital money. All the things are not gonna be able to work together for a very long time. Unfortunately, I expect that this is gonna be pretty tough here in data infrastructure in 2023, perhaps 2024.

By all means, I hope I'm wrong. I hope the macro environment comes roaring back. I have all sorts of companies that would benefit from that. So, by all means, I'm with everyone on that. But I think we are at the beginning of a more Darwinian period where in each category there's gonna be one or two companies that survive and then a bunch of companies that are not; and when I say they're not, it doesn't mean they necessarily go bankrupt.

Unfortunately, some of them will. But I think it means that a lot of companies get gobbled up very often for not very much money and not generating the kind of return that the founders, the employees, and the investors were hoping for. But, I think we are at the beginning of this trend as opposed to the end of it. I think for a lot of those companies the moment of reckoning has not even happened yet because so many companies raised a bunch of cash in 2021, and they still have, year, two years, sometimes three years of cash.

So they don't really have to worry about the immediate reality of what raising a round will entail. But I think that moment is gonna come at some point, and when it comes, that's when it accelerates everything in terms of " okay, our business is not working. We can't raise another round, therefore we need to find our home. Oh shit, we cannot find our home, therefore we are going to have to go and just move on and do something else."

And look, I think a lot of those companies are just early, right? And, a bunch of companies were started in 2019, 2021, so they're one, two, three years old. And it's just not a lot of time to build a business, even with great founders, even with a smart positioning of the product, even with great execution, even with venture capital money. That's a little bit of a time on Earth dimension that matters immensely in building a company. And It is just not enough time to get the kind of escape velocity that will enable you to raise more money and therefore be able to survive in the top market.

So again I hope I'm wrong. I hope somehow it all works out. I have a lot of friends in this industry. There are a lot of people I deeply respect in this industry and all the things. But think we're gonna have a tough couple of years ahead of us.”

We are currently seeing an evolution away from the “Modern Data Stack” within the data market. Given this change, what type of data infrastructure startups are you most excited to invest in?

Matt: “Yes, I ask myself that question a lot these days. So I do agree that there's at least chatter about evolving away from the Modern Data Stack. Although I think the reality is that it's gonna be a lot more nuanced than this. But, like everybody else in conversations, I'm hearing that the whole idea of paying an ELT vendor, ETL vendor for a lot of money, and then you data warehouse a lot of money, and then the transformation layer a lot of money, and the visualization layer a lot of money, and then like stitching everything together, and that's expensive and time-consuming.

Also hearing that general philosophy, which has been one of the core tenets of big data since the Hadoop days, that taking all your data and dumping it into a repository and worrying about what you're gonna do with it later. That's getting certainly under scrutiny because as it turns out, it's technically doable because, Snowflake is super elastic and all the things, but still very expensive and not always that useful. Something I'm seeing people for the first time since the Hadoop days maybe that's not exactly the right approach. So all of this is definitely happening.

Now, in terms of companies I'm excited about there's certainly a generation of companies that seem to be sort of taking a different stance. Like the perfect example that has been obviously very buzzy is DuckDB. And this concept of disaggregating it all up if you want. So I guess it was like one central thing, like doing it in an embedded manner. That's certainly interesting. I think despite the buzz, and I'm perhaps sadly not an investor in the company, but from what I'm hearing is still pretty early and a lot to prove and all those things. But in terms of approach, I think that's really interesting.

There's a whole different category of companies where I was not sure at first, but I actually do think it's interesting, they seem to be getting quite a bit of traction which is the rise of the fully managed platform. And I'm not an investor in those companies, I don't know the details yet, the Y42, Mozart Data, Keboola and they do things in different ways, right? The Mozart Data and Y42, as I understand it, just take all the usual suspects and stitch them together and abstract where the complexity where as the Keboola has built all the tools themselves natively. But regardless, I think that approach of " okay, you have one platform that does everything regardless of how you do it." I think that's interesting in a context where you start focusing more on just simplicity and everything working and you may have like less resources on your end to do experimental stitching together. So the disaggregation of all-apps-on-one and then the rise of fully managed, I think those are two interesting trends.

And then maybe a little bit, the periphery of the Modern Data Stack, I'm excited about the general evolution towards more sort of convergence and more simplicity. Precisely when you look at the MAD, you have tools everywhere that you also see different things. And it seems whether that's at the database obstruction layer or in other parts of the ecosystem, that things seem to be converging toward another.

If you look at the database abstract space, which is what I call it, it may not be the exact term, but I recently invested in a company called SurrealDB which is an abstraction layer on top of a key-value store. And that does a lot of things that do like document and graph and like real-time and it's serverless and so a lot of the things that people have been talking about for a while, that has been historically very hard to combine. That's now happening. So I think that's pretty interesting.

I was also an investor in CockroachDB, which is like more SQL with some of the capabilities of NoSQL in terms of scalability. But that's also like simplifying that complex problem space and doing different things that people didn't think could be combined together. So I think those are interesting spaces.

And in the same theme of convergence. It's weird that as a space we had like the people that do the batch things and then another group of people that do the real-time things. So I'm interested in that convergence as well. And I'm an investor in a company called Estuary. That at its core, does a lot of this "why do we need to have real-time and why do we need to have batch? How can we just unify everything together and rethink our ELT in that general context." So that convergence and that trend toward conventions and simplification feels inevitable to me and very interesting.”

You are the organizer and host of Data Driven NYC, one of the largest communities focused on data, ML/AI, and enterprise software. What has been key for you in both growing and maintaining such a strong tech community over the past 10+ years?

Matt: “It's gonna sound terribly self-serving, but the love for it has been the key driver because it's very easy to do one or two events, or four or five. It's really hard to do it over several years and even harder to do it over ten years, which is what I've been doing at Data Driven NYC.

And especially there's an offline component, which is a big part of Data Driven, which has pros and cons, but like ultimately I think is very powerful. Every event you need to get butts in seats and you just constantly just rebuilding the event each time; like look, this is not show business, but you are literally only as good as your last event.

And so the consistency of the effort required I think has been the key driver for doing this. And then that's on my side. And then I think on the community side, I think you wanna create that flywheel where great speakers bring a great audience and keep great speakers coming back.

And it's not just a number. I think the biggest surprise to me for Data Driven over the years has been the sheer quality of the people that show up with the things. And there are again and again people that could then maybe will be on stage in the future. It's just I guess that's the beauty of doing something that's pretty geeky. Like you get a self-selecting group of people who shows up again and again at some meetup to talk about like data and machine learning or AI; like people that are irrelevant to the space will go once, but they won't come back. But the people that keep coming back tend to be of a level of intellect and passion and just knowledge of the space-- sophistication. That makes everybody's experience just incredible, including the speakers.

Over the years, speakers have commented again and again about the quality of the discussions that they had, and it's every speaker's fear that after you've done your video panel or your presentation, you get swarmed by people that are just wasting your time. And you know what, what I heard over the years is that it's very different from their perspective just because people are just good in the audience in general.”

Person Profile:

Matt Turck is the Managing Director at FirstMark Capital. Feel free to connect with him on Twitter and LinkedIn to learn more about his work.

What are others saying in the DataOps space?

The 2023 MAD (Machine Learning, Artificial Intelligence & Data) Landscape

• What: A dizzying display of the various vendors in the data industry that just grows every single year… this is a lot of work to compile, so it’s awesome that our industry has this.

• Why: An opportunity for you to validate that you have indeed heard about a new data tool every other day.

• Who: Any data professional interested in a snapshot of our industry.

MAD 2023, PART II: FINANCINGS, M&A AND IPOs

• What: An overview of the funding side of our data industry for the past year.

• Why: A great discussion on how data companies can adjust to our market potentially headed towards a downturn.

• Who: You are a founder that is looking to raise funding and wants a reality check.

MAD 2023, PART III: TRENDS IN DATA INFRASTRUCTURE

• What: A dive into how data infrastructure is starting to change in our industry (e.g. bundling and consolidation).

• Why: This is one of the best summaries of emerging trends in the data infrastructure space.

• Who: You are a tech lead or head of data platforms trying to understand what your data stack could look like in a few years.

MAD 2023, PART IV: TRENDS IN ML/AI

• What: Have you heard of this thing called ChatGPT? Of course, you have!

• Why: Even though we have heard a lot about ChatGPT as a tool, few are talking about the market implications of generative AI… this article is a great primer for understanding its impact.

• Who: Anyone who is trying to make sense of generative AI and its potential impact on their business and or career.

Bonus: Very timely is the situation of Silicon Valley Bank and its impact on the startup ecosystem. Below is a great tweet thread describing the situation.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

What are your thoughts on acquisitions in the data market?

We are in a drastically different market from even a year ago when it seemed like our industry was being pumped up with unlimited VC money, and every day a new data tool popped up. Fast forward to today, and leadership teams are asking departments to do more with less. This reduction in potential revenue for startups leads many to ask “what if major players started acquiring these startups?” Maybe we could finally implement the Modern Data Stack without the need for 10+ vendors— one vendor to rule them all! Jokes aside, my conversation with Ethan Aaron below explores this question, but his quote from the interview best summarizes the answer: “It's very difficult to spend a hundred million dollars…”

-Mark

Announcement:

I’m speaking at two upcoming conferences!

DSCO 2023, Data Science Conference, March 12 – 14, San Francisco

• Session: Data Science as an Equalizer

• Information: https://dsco.usfdatainstitute.org/data-science-as-an-equalizer-1

Low-Key Data Conference, 3/22 from 1-5 pm EST, Virtual

• Session: TBD

• Information: https://tinyurl.com/4tydj8km

Hear from Ethan Aaron, Founder & CEO at Portable:

Hear from "XYZ" highlights real-world use cases for all of us to learn best practices and upcoming trends within the DataOps space. I first talked to Ethan last year as a networking call to learn about a fellow data nerd, and our conversation has stuck with me since. Specifically, Ethan comes from a background of both leading a data team and finance (Mergers & Acquisitions), giving him a unique perspective of the data marketplace. In our first conversation, he shared how he believes many data companies have raised too much capital and its impact on the future. I haven’t stopped thinking about this, so I had to bring him onto the newsletter so you all can learn as well. Enjoy!

How did your background in mergers and acquisitions within the data industry prepare you to be a founder?

Ethan: “I think there are a couple things about having a background in strategy and M&A that are really valuable to being a founder. Number one, if you ever work in M&A or finance, one of the best things you get very good at is creating landscapes of the ecosystem. You aren't looking at a feature. You're not looking at a user. You're looking at the entire landscape. All the companies that can serve a certain use case. And I just happened to be looking at the data integration ecosystem. So I'm coming into this with a very good understanding of the ecosystem we were about to play in.

Luckily, the M&A experience gave me a very good understanding of the ecosystem, but even better for me as an ELT company, is before I worked at M&A, I stood up the modern data stack. So I went into my research in the industry from an M&A perspective, having first done this myself. I was talking to companies, being like, "Hey, I need to put in place a data warehouse. Hey, we're looking at ETL tools, data collection tools, visualization tools..." we had a bunch of these tools in place in various parts of the organization at LiveRamp when I stood up a centralized data team.

Number two is capital structures. Something a lot of founders don't have a ton of understanding of when they start their company is what are the implications of raising capital; whether it's debt or institutional rounds, or safes, or taking money from angels. What does that mean for future expectations, growth expectations, and what does success and failure look like? And looking at companies from the other side of the table, be like, "Hey, I wanna acquire your company." the first thing that comes to mind is the capital structure. Can you afford the company? Will their investors allow you to acquire them, et cetera?

So I was able to go look and use these tools first, then I worked at M&A, and then we started the company. So, of the experiences, the M&A experience provided landscape understanding and capital structure. But honestly, the experience running the data team is when I talk to our prospects and clients like I've been in their shoes before and I'm in their shoes now.”

You have shared that many data companies are over-valued within our fractured market. With the changing market, what does this mean for consumers considering vendors?

Ethan: “Thinking about it through the lens of a consumer. So number one, let's rewind back to 2021 and 2022. Interest rates were very low, which means valuations of everything were very, very, very high. You could have one dollar revenue, and the value of your company was a thousand dollars. We saw the same thing happening in the data world. Companies with very little revenue saw unbelievably high valuations, unicorn-level valuations. Or companies with menial revenue having very, very large multi-billion dollar valuations.

What does that mean? It means that those companies raised a lot of money. So if one of these companies raised a hundred million dollars, two-hundred million dollars, three-hundred million dollars, you don't really have to worry about these companies disappearing overnight. That's not the problem that is going to face companies that took on money at too high of an evaluation. So as a consumer, as long as the companies have capital, I wouldn't worry about are they gonna disappear.

It's very difficult to spend a hundred million dollars, so I wouldn't worry as much about that. My hunch is that who's in charge of those companies might evolve over time. When you don't have institutional investors that own most of your company, the founders and the CEO have a lot of power over the strategy, the product development, the hiring, and all that.

When you raised at unbelievable valuations and the board is not the founders, it becomes a question of how do the investors most effectively get a return on their investment? Hopefully, that aligns with the founders and the CEO and the people that are running these companies, but I would say as a consumer, as the person actually buying products, that's the dynamic.

The companies are not gonna go away, but inside of these companies, a lot of employee equity at this point is probably worthless. It's underwater. So are the employees all gonna stay? Maybe, maybe not. Maybe they have to get new equity packages. Are the founders gonna stay? Again, maybe not. If the valuation was too high and you can't get back that, their founder equity might not have value in it, and the board might want someone else to step in and try and create value. So I think consumers double-check to make sure these companies will be around. And I would expect slightly less innovation, new features, new products, because of the turmoil that could exist in the coming 24 months.”

Given your experience in finance, how are you best positioning your company to be competitive within a down market?

Ethan: “I would say there are a few different things. So up until now, we've been unbelievably lean, and we're not doing that necessarily from a financial perspective. We're doing it from an operational perspective. We believe the problem Portable solves is an operational problem. We wanna build 10,000 integrations.

Money doesn't solve that. If you gave me a hundred million dollars and told me to go hire all the people to build 10,000 integrations, I can't do it. What we need to do is figure out how to build a platform to get there. So our ability to stay lean has been in service of that vision and that mission. How do you operationally scale your business? Not how do we save money.

That being said, when I think about finance, everything's an investment. You put money in and you get an investment, you get a return on it. What that return looks like changes. So one of the things I think about right now, and I think about it like portfolio allocation, is go-to-market for Portable.

There are ten different channels you can sell your product through or service through. One, enterprise sales, hire an enterprise sales first, and they have 50 meetings with a company, you sign a hundred thousand dollars. Option two, SEO, search engine optimization. You write a bunch of content, put it on your website, and wait.

It takes a lot longer. It drives views, which overtime drives clicks, which overtime drives signups, and overtime money. So the return on it is much slower. But the economics of search is pretty remarkable as well because you can write a hundred pieces of content and then forget about it, and it will always effectively pay a dividend.

You can do paid ads, pay $5,000 this month, and get traffic to your site. The problem with that one is it's fast. You get signups now, but if you shut it off, it's gone. So when I think about a lot of the decisions we make as a company, whether it's investing in profit or investing, go to market channels. I view them all as investments.

Right now, the couple investments we're making, one SEO, is that gonna pay off in the next two months? No, not at all. It's a long-term play. So either our portfolio of go-to-market investments, it is something that we are gonna want to have done a year from now. So we're starting now.

We also need faster returns. We need signups today because those signups today turn into advocates that start telling the market Portable exists and that they love the product. So when I think about finance, there are all the capital structure aspects. How do you raise money? How do you tell stories of investors?

But one of the most interesting ones is every decision you make, every person you hire, every feature you release... how much of the cost, and when does it return that investment? If you can't get those numbers to work, don't make the investment. Some numbers are long-term, and some are short-term. But I think the biggest thing is just to think about every one of these decisions as capital or time goes out, and then over time, you either need to recoup capital or time. So that's the biggest lesson learned and kind of framework that I've been using a lot recently.”

Person Profile:

Ethan Aaron is the Founder & CEO at Portable. Feel free to connect with him on LinkedIn to learn more about his work and to check out his company.

What are others saying in the DataOps space?

‘It’s All Just Wild’: Tech Start-Ups Reach a New Peak of Froth

• What: A New York Times article from January 2022 that best captures the state of funding in tech.

• Why: This article provides historical context to my interview with Ethan Aaron.

• Who: You are a founder or VC who wants to remember the good ol’ times before capital became expensive again.

The ‘Too Rich to Succeed’ Challenge Facing Start-ups

• What: A 2014 article from Wharton describing how a surplus of money combined with high investor control can lead to startups failing.

• Why: This article also provides historical context to my interview with Ethan Aaron.

• Who: You are a seasoned tech veteran experiencing a deep sense of deja vu.

Modern Data Stack: Use Cases & Components (2023)

• What: An article from Ethan providing a great overview of the Modern Data Stack.

• Why: There are so many tools that can be included in the Modern Data Stack that it can be hard to grasp what exactly it is.

• Who: You are looking to implement a component of the Modern Data Stack and want to review potential solutions.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

What are your thoughts on AI?

The major shift in AI is surprisingly not technical… it’s cultural. ChatGPT, for better or worse, has brought AI to the masses in that it is 1) tangible, 2) can be experienced by anyone, and 3) a 10x experience from standard processes. Everyday people outside of academia and tech circles are now wrestling with the externalities of AI:

• "What happens when AI takes over white-collar jobs?"

• "What is the role of copyright when content is AI generated?"

• "How do we account for racist and misogynistic training data influencing the AI being used by the masses?"

Questions like these were always present, but ChatGPT now forces society to confront them... for better or worse.

This shift became apparent to me when I attended the TransformX conference last October, where I heard leaders such as Greg Brockman (Co-Founder of OpenAI) and Eric Schmidt (Former CEO of Google) speak about the future of AI and foundation models. In summary…

• We have “opened Pandora’s box” and must face both the positive and negative consequences of this powerful technology.

• Similar to how the iPhone and App Store created new business models and markets, foundation models will enable innovative businesses to build off them.

• There is a technical arms race between international powers to produce the most efficient chips and thus take the lead on AI.

You can check out the recordings from this conference via this link.

— Mark

Hear from Jazmia Henry, Senior Applied AI Engineer at Microsoft:

Note: Jazmia informed me that the recent Microsoft Layoff unfortunately impacted many of her AI researcher colleagues that she directly worked with. Please reach out to her if you are looking to hire talented AI researchers, as she will be happy to connect you. You can learn more from her recent post on Linkedin.

Hear from "XYZ" highlights real-world use cases for all of us to learn best practices and upcoming trends within the DataOps space. When I think of people defining what AI technology looks like in the future, my friend Jazmia quickly comes to mind. She has led ML teams within finance, contributed to AI research within top universities such as Stanford, built products exploring edge computing on the blockchain, and is now an AI researcher at Microsoft. Her insatiable curiosity about technology, its future, and social impact is a driving force for her work and what makes me so excited about the interview below. Enjoy!

Subscribe now

What are the unique data challenge you face as an AI researcher working within reinforcement learning?

Jazmia: “So reinforcement learning requires you to know two things. One, to have an overarching goal because you want to ensure your simulated environment is as close as possible to wherever you're going to deploy the model. What would traditionally happen before I would go into reinforcement learning is, let's say, I would have a bunch of unlabeled data. I'm trying to figure out what direction to take this unlabeled data. You can sometimes do something where you go, okay, I'm trying to, for example, let's say, cluster my customers into different groups. You can just kind of figure out where the data's taking you, and then just kind of follow it, down wherever it takes you, and then deploy it once it's good enough.

In reinforcement learning, you are training an agent to understand a physical space using a simulated environment. So whatever environment you create has to be as close as possible to the space you're deploying it in. So that requires way more time and effort collecting appropriate data.

And many times there is no appropriate data… it comes to a problem, and you're like, we want to create this model that's going to help us create the best Cheetos, which is an example at my company as we work with PepsiCo. I wanna create the best Cheeto possible. Well, where am I gonna find data on how to make the proper Cheetos?

I have to make that data. I have to create that simulated environment. The environment has to be as close as possible to what the actual lab is like, right? So I'm gonna spend six months collecting data and finding experts and talking to these experts so I can create that environment.

You're replicating it in a way that acknowledges just how messy humans are. It's really like when you're building a machine learning model, you want the data to be clean. But when you're building a reinforcement learning model, you don't want the data to be clean, like you want it to be almost as messy as real life is.

Because let's say I collect a bunch of data at some lab making Cheetos in Texas. It has a level of humidity in there. Maybe it has people who are more likely to be walking around in a certain way that might be different than how somebody might walk around Boston. Maybe in Boston, it's colder and less humid. People are less likely to run up and down the halls, not because of any other reason other than maybe just cultural differences. The type of people that you have in the office might be different type of people. So how will I have a machine that's working with a human being that’s going to be different than expected?

Because if I know that Joe, who works in a factory in Houston, is more likely to be clumsy, I gotta make different adaptions than an office in Boston, where maybe I might not have Joe who's clumsy. But maybe I might have Dave, who's likely to move the machine outta the way and do something himself, right?

I'm creating different spaces. And I want my data to train my agent in a way that's adapting to the difference in space. So something that might be like, “oh, this is crazy. This is a weird observation. I'm gonna just drop it from my data analysis.” You're not doing that in reinforcement learning.

You're like, “this is a weird thing I'm seeing. Let's train on it because that might be something that, later on, will be important to help us continue with our process.” Maybe their machine's running hotter in the South. I'm gonna need to keep that in my data analysis.

So that's the biggest challenge, having a goal and that goal being as close to the actual problem, especially if you’re gonna deploy it. And then the second thing is finding that darn data and making it as close as possible to the actual world so your agents actually learn something of value and not fail.”

Through our previous conversations, you have described to me your deep interest in deploying AI on distributed edge devices. Can you elaborate on what excites you the most about this future of AI and why more people should pay attention?

Jazmia: “So much of AI is what, at least it's coming out, has been coming out in the past few years. I would consider it to be disembodied AI. I don't think that's a proper term. That's just a term that has made it make sense to me in my head.

And what essentially that means is your ChatGPT, for example. You go, and you put in some information into your computer, and then that API, there's a call to the actual machine and then back and forth, right? It's not making any actionable decisions in physical space at all. I cannot say to ChatGPT, "Hey can you teach me how to create a great cake?" And have it reach out and begin grabbing ingredients. Instead, it's just going to give me a list of ingredients. Oh, you might wanna have some flour. You wanna have X, Y, and Z. Those things are cool, and they're interesting things, and they're important things, and they are steps that eventually could lead to what I would consider being embodied AI.

Which is AI that has a physical space. Where I'm able to say, "Hey, machine, can you teach me how to make a cake?" And it's actually teaching you how to make a cake. It's showing you the decisions it would make in a way that you would have to do... and those machines would be distributed. Those machines would be devices with some type of internet of things capability so that the API call can be within some type of edge device. I think things like that would be really cool for a couple of reasons. But the biggest reason, it's gonna sound so silly, is because this is a type of AI that not only I but also most people are used to, even though it's not commercially popular.

So when you were a kid, and you were watching a movie, and they were talking about some type of machine that was really smart and intelligent, it was a bot, it was an android, it had a body, it was doing things, it was reaching for things. Whether you are watching the Jetsons with Rosie the robot, or iRobot with Sunny, or Ultron, right? They were able to do things and have conversations with you and make adaptions and be funny and things like that.

We haven't gotten really to that space yet when it comes to commercial AI. But naturally, to me, we would have to progress into these spaces because when you talk to people about AI, that's what we think about. Especially if we don't work in this space, we're thinking about what we saw on TV, which had some type of physical component. And so, to me, that is a natural progression, being able to create AI that's not only intelligent, smart, able to adapt, but it's able to have some type of physical space with us and work with us for a better, more equitable future.”

You have experience as an exceptional individual contributor and leading exceptional data teams. What advice can you give leaders on fostering teams where technical ICs can thrive?

Jazmia: “First, have a vision for your team and have your vision be compatible with the team that you currently have. I sometimes think what managers do, especially when they're new to management, are they come in and they don't have a vision other than, “oh, we are gonna be the best team,” but they have like no steps of what that looks like.

And then if they do get to the point of having an idea of, okay, in five years, what are we doing? And this is gonna sound so horrible, but I usually found many of them will describe their teams… and I'm like, "that's not your team." You're describing to me a team that's able to build reinforcement learning models and can deploy them to some type of edge device, but then I look at their team, and it's full of analytics folks… Nope, not gonna do that with that team.

And so, the problem with doing that, with not marrying your vision to your team, is that a) you're never going to get to that vision, and b) you kind of create this idea to your team that they have to do extra stuff to be part of that vision. Now they're trying to fit into your vision versus you having a vision that's appropriate for them to grow.

And so, if you're managing a team, what you wanna do is you wanna look around at the skill sets that your team has. You want to have conversations with them about what makes them interested, what gets them up in the middle of the morning pretty excited to go to work. And you want to make sure that whatever projects and proposals you're putting forward, or ones that are compatible with their hopes and dreams.

In my last company, when I first started leading, I had a vision of leading a bunch of quant-based data scientists. That's the space that I came from. And then we started hiring people, and most of them were machine learning engineers that didn't have quant backgrounds— so they had a way of doing things that were different than how I did things. So now I'm learning from them what type of work they find interesting and what gets them up in the morning. And it's different than what I thought we would be doing. And so we just made adaptions, and we changed our visions to be compatible. So if I had my leader say, “oh, you're good with natural language processing, let's build a chatbot.” We could do that. But my team, no, we don't do that. So instead, let's build some recommendation engines that can do X, Y, and Z things because that's what my team enjoys doing. That's the type of work that's good for them. So that's what you wanna do as a leader.

And then this one is a little bit gratuitous, but I think it's a good add-on. If you want to have your IC thrive, you have to trust your team. There is no reason, especially in the land of data and data engineering of any type. There is no reason for me to be upset because it's 10:05, and I look on Slack, and my IC isn't signed on. If there's no meeting at 10 o'clock and they're not signed on at 10:05, let it go, right?

You don't have to have them doing eight to five or whatever. There's no point in that other than micromanagement. If there are no meetings that they're missing, leave them alone. People have kids, people have lives. Most engineers work at home. So be acknowledging of that.

And when you do that, you're actually gonna get your best work from your engineer, cuz they're not gonna feel like. I gotta create some BS bug to stay online long enough for my manager to feel satisfied. Instead, they're just doing the work that they have to do. And you know they'll give you ten times what you asked for because they know that you trust them, and they end up trusting you back in turn.

Many leaders who are very well-meaning have no idea what it's like to be a data person. Data is very new. And so most of the time, people who come from data land haven't yet made it to the point where you can get up to being executive. And so, if you're going to manage your team, advocate for them. Don't be afraid to go into leadership and say, "Hey, I understand that we have this structure where we're telling people they have to do X, Y, and Z. That's great for the rest of the company. Let me tell you why this might look different on my team."

A lot of times, especially when you're talking about startups, that's what makes people stay more so than simply giving them a bunch of perks and ping-pong tables and stock options. Those things are nice, but them knowing that their manager has their back, it's going to do wonders for them.”

Person Profile:

Jazmia Henry is a Senior Applied AI Engineer at Microsoft. Feel free to connect with her on LinkedIn to learn more about her work.

What are others saying in the data industry?

Below are some select articles written by Jazmia.

MLOps: A Primer for Policymakers on a New Frontier in Machine Learning

• What: “An explainer of tools for bias mitigation in the MLOps lifecycle.”

• Why: It’s a deep dive into how ML models are deployed and their impacts on their outputs.

• Who: You are interested in how we can make the future of AI equitable.

On AI and Types of Reasoning

• What: Jazmia shares the parallels between AI and the way humans think.

• Why: It’s an approachable read about how AI “thinks.”

• Who: You are interested in the ways in which we can use AI to reason through various problems.

Model Rollbacks Through Versioning

• What: A technical deep dive on how organizations can save a substantial amount of money on their ML deployments.

• Why: A clear explanation of versioning ML models and their impact on deployment costs.

• Who: You are a leader looking for ways to reduce your spend on ML initiatives.

About On the Mark Data:

On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.

I was recently asked about DataOps on a podcast interview, and I couldn’t stop referencing my interview with Christopher Berg! His interview was my first newsletter edition, and thus few people have seen it— it only has hundreds of views compared to the thousands in my recent editions. In other words, this is a hidden gem that I think more people should be aware of. Christopher Berg and his team have played a huge role in making the data industry aware of the value of DataOps and creating resources to help people learn about DataOps.

Finally, I also want to highlight someone else you should be paying attention to in the DataOps space! , who is a data platform engineer and the author behind the , has been writing some amazing pieces on the topic. Below are three I highly recommend:

Dutch Engineer’s Newsletter

DataOps' Components

DataOps and data engineering are distinct practices, even though DataOps stems from data engineering. Data engineering focuses on the technical aspects of data infrastructure, such as storage, pipelines, and data movement from source to target. In contrast, DataOps is a methodology that encompasses the underlying principles and processes that support an…

Read more

4 years ago · 2 likes · Sarah Floris

Dutch Engineer’s Newsletter

6 Hands-On Techniques for Optimizing Data Operations

I did not realize, initially, how important it was to incorporate clean code practices into my development process. However, as I have progressed in my data engineering career, I have come to understand that following DataOps principles is a key step in the right direction. Building out these practices can be a time-consuming process, but it is well wor…

Read more

4 years ago · 3 likes · 2 comments · Sarah Floris

Dutch Engineer’s Newsletter

Common Obstacles in DataOps

As a Data Platform Engineer, I can vouch for the many benefits of DataOps: better collaboration & coordination, automation, fewer errors, security, best practices, reusing existing materials, self-service capabilities, and data democratization. Unfortunately, before we see these benefits, we have to implement DataOps and overcome the following obstacles…

Read more

4 years ago · Sarah Floris

What are your thoughts on DataOps?

Between MLOps and DataOps, it may seem like X-Ops is the new trend in the data space— but make no mistake— these concepts are rooted in years of data practitioners sharing how they solved real data problems within businesses. DataOps was first coined by Lenny Liebmann in 2014 as “the discipline that ensures alignment between data science and infrastructure.” Though the Modern Data Stack has taken up the most mindshare since then, the pain points presented by DataOps have only grown to show its necessity. Specifically, the recent rise of data engineering and the push toward data-centric AI signals DataOp's importance in delivering high-quality data to customers and internal business stakeholders.

Hear from Christopher Berg, CEO, Founder & Head Chef of DataKitchen:

Hear from "XYZ" highlights real-world use cases for all of us to learn best practices and upcoming trends within the DataOps space. When I asked my network who I needed to talk to regarding DataOps, Christopher Berg was repeatedly mentioned. It became clear, after looking further into his profile and reading the DataOps Manifesto, that Christopher is leading the charge for the DataOps movement.

What is DataOps, and why should organizations care about it?

Christopher: “Yeah, I guess kind of two related reasons. One is like if you actually do the work in data science or data engineering, your job sort of sucks, honestly, because you spend a lot of time doing things that aren't really delivering value. You, you have a lot of failure. A lot of people are quitting and being upset.

We did a survey with 700 data engineers with data.world last year, and 78% of data engineers were so stressed they wanted a therapist on their job, and so you're caught between this kind of, ‘I gotta work really hard,’ ‘I'm always behind,’ ‘my customers always want new stuff,’ and then it's breaking left and right.

And so your life sort of sucks in a lot of ways. If you go on the other side, like you look at the people who are really trying to influence with data, right? Business people, maybe people on your website, they're dissatisfied as well. They want more. They don't understand it. And so there's all this potential around data and all this potential to sort of help and change.

And the people who are kind of working to make that happen are very unhappy and, and the results aren't there. Most projects fail, most models don't get in the production. And so I think that's really what we're trying to address with DataOps is that sort of failure and pain that people have doing their work.”

An organization’s data maturity falls on a spectrum, at what stage in the data maturity journey should an organization start taking DataOps seriously?

Christopher: “It depends on how much you like being a hero. Like all the people say, ‘wow, you worked all weekend,’ ‘you're amazing,’ ‘I love you.’ And like, how long can you be a hero? Right? And, and what happens with people is if they don't start thinking about building a system to make their life easier where they can test and automate and observe and iterate, they get burnt out and then they end up honestly quitting.

And so we all wanna do the cool things, but if you're doing the cool thing as a hero, no matter if your team is big or small, no matter if you're at the start of a project or the end, you're creating problems. For me, when I was younger, I spent a lot of time kind of being the hero and the project, and I left a lot of sort of hair balls for other people to pick up, and it's not cool.

You write a bunch of code, it's untested, it's un-version control, you change it right on production. Every software engineer goes, ‘ew,’ and we do the same thing in data, but no one goes ‘ew.’ And so I just want everyone to collectively go ‘ew, you, you're hair balling it.’ And that's my quest, to get the entire data community just start doing that collective ‘ew’ that software engineers do.”

What's something that you believe the broader data industry is missing about DataOps, but should care more about?

Christopher: “Well, I think it's observability led DataOps. So I've come to believe that we are, as a company DataKitchen, not gonna change right now how people build things. They've already built things, they're already being heroes. I've been saying the same message now for six years and the world hasn't changed.

And so what I'm saying first is observe what's happening with your system. Stick little thermometers at various points in the process and measure is it running, is the data right, is the model still predicting, and centralize that information. And you're gonna be really surprised at what you see.

That information, that data is gonna drive your behavior change. If you can measure errors, either from poor code that's getting in the production or poor data that's getting into your system, if you can see that you're late, if you can see if your system's utilized. That source of information is actually really insightful.

So observability first, get the information, stick a bunch of thermometers all over your data pipelines, your models, your viz, measure all that stuff. Then look at the data and say, ‘where are the bottleneck, where are the errors? Let's just automate them.’

And to me, I think that's gonna be the gospel we're talking about first because people are gonna continue to hero out and build these systems and they're in production, they don't wanna change them, they're afraid to change them. And so once you stick their monitors in, once you start saying the problem, you'll say ‘oh wow, maybe I should put some automation on it, maybe I should develop some more unit tests or system tests, maybe I should figure out deployment and version control.’ All these things will start to come.”

Person Profile:

Christopher Bergh is the CEO and Head Chef at DataKitchen. Chris has more than 25 years of research, software engineering, data analytics, and executive management experience.  At various points in his career, he has been a COO, CTO, VP, and Director of engineering.  Chris has an M.S. from Columbia University and a B.S. from the University of Wisconsin-Madison.

Chris is a recognized expert on DataOps.  He is the co-author of the ‘DataOps Cookbook” and the “DataOps Manifesto,” and a speaker on DataOps at many industry conferences.  Chris began his career at the Massachusetts Institute of Technology's Lincoln Laboratory and NASA Ames Research Center. There he created software and algorithms that provided aircraft arrival optimization at several major airports in the United States. Chris served as a Peace Corps Volunteer Math Teacher in Botswana, Africa.

What are others saying in the DataOps space?

DataOps: Why Big Data Infrastructure Matters - Lenny Liebmann

• What: An early warning of the limitations of big data only focused on data science.

• Why: See the origins of the DataOps movement as defined by Lenny Liebmann.

• Who: You like to understand how we decided on specific frameworks within technology.

What is DataOps? - Tamr Inc.

• What: A reflection of DataOps for the past seven years regarding the trends driving the movement and the difference between DataOps and DevOps,

• Why: The author, Andy Palmer, was one of the earlier voices in popularizing DataOps.

• Who: You are trying to understand why DataOps is important and how it compares to DevOps.

Download The DataOps Cookbook | DataKitchen

• What: A ~200-page, in-depth book of “methodologies and tools that reduce analytics cycle time while improving quality.”

• Why: An excellent reference to guide your journey into DataOps.

• Who: You have moved beyond learning about DataOps and want to start implementing DataOps.

What are your thoughts on strategy?

Data leaders often talk about the importance of strategy, yet many fall for the trap of confusing strategy with tactics. The three guiding questions that help me maintain a strategic mindset are the following:

1. How can data create a competitive advantage in the market, where the result of the initiative enables a clear differentiation from competitors?

2. What are the high-touch human tasks that drive revenue (e.g., enterprise sales), and how can data optimize the key points leading up to these high-touch tasks?

3. What assumptions drive my strategy, and how can I incrementally validate them to reduce the risk of implementing one strategy over another?

These questions were formed through my numerous conversations with Vin over the past few years via Zoom office hours, strategy courses taught by Vin, and even one-on-one career coaching sessions I have had with him. Even with all of these conversations with Vin, I still feel like I’m only scratching the surface of his knowledge on the subject— hence why I’m so excited for you to read my interview with him below. Enjoy!

— Mark

Hear from Vin Vashishta, Founder & Technical Strategy Advisor at V Squared:

Vin ranks as one of the top three people that has helped elevate my data career with his wisdom. I specifically attribute his data and AI strategy course to helping me get promoted to a senior data scientist and driving tremendous value at my job. In addition, I used to have near-weekly conversations with him on the Artist of Data Science Happy Hours, where he would drop gem after gem of knowledge, which slowly pushed me to be a more strategic thinker. In short, I’m beyond excited for you all read the below insights from Vin!

One of my biggest takeaways from our previous conversations is the need for data teams to be positioned as strategic partners rather than cost centers. What do you believe is necessary for a data team to be positioned as a strategic partner?

Vin: “We have to have revenue booked against our names. If cost savings are great, productivity is great. That's gonna get buy-in. That's sometimes what you're stuck with in the early days. But if we don't have revenue, there's no point to our existence because we're too expensive.

Software development teams are the paradigm companies are used to. That's the cost structure they're used to. When they start digging under the covers and CFOs are, believe me, they're lifting up the curtain right now and they're looking behind and seeing what's going on. They realize how expensive in comparison to software engineering that data teams are.

It's the people that are more expensive. It's the infrastructure that needs to be built out that's more expensive. Data gathering just by itself is more expensive. We're going to quickly shift from data gathering to data curation, but only after, as a data science team, we start showing a value proposition that grows the business. But that's the interesting piece of where we're going.

Companies are going to start seeing because now they're forced to monetize data. Startups need a path to profitability. That's gonna be one driver for this trend. Companies that are struggling to remain competitive need new growth areas; they need to expand into new marketplaces. So in big tech, those companies that are struggling for growth, to maintain their share price and maintain their multiples, they need ways to grow. They're going to turn to data. It's gonna be another driver of this.

And in traditional legacy businesses, traditional industries, those companies have to figure out how to maintain margins in the face of the current inflationary cycle. But longer term, they're now challenged by tech companies who are going to come into their industry and begin to compete with them on best in class technical capabilities.

So those are your main drivers of what comes next. Data teams either step up or their businesses are going to fail. So you'll see these use cases over the next year start emerging more and more where a company that made maybe 40 million, 50 million in total revenue in 2022 figures out that they have a data gold mine. And by the time 2024 comes around, they're in 500 to a billion. Because that's the power of data. It is truly insane how much money you can make off of a few initiatives. But you have to do things in a more strategic way. The data team needs to put itself at the strategy planning level in order for opportunity discovery to be done properly.

We have to have an entirely different conversation with the C-Suite to explain how data creates value differently than digital products create value. There's a different life cycle, a different process for management. It's going to take a company that's desperate. Or a company that realizes the absolute just blue ocean gold mine that's in front of them.

If C-Levels realize that, the company will quickly grow and it'll be one of those early use cases. If it doesn't, it will not survive. So that's going to drive this trend over the next 12 months is this realization of risk and peril as well as opportunity and what it will take. But once we get a few of those use cases out in public, it will be a land grab for data.

Dive deeper into this response with the following suggested reading from Vin:

High ROI Data Science

What Mark Zuckerberg Can Teach Us About Selling Data Science Initiatives To C-Level Leaders

Mark’s right, but he talks to businesspeople like they’re engineers. He’s a lesson in public view about why data science fails to capture the attention of C-level leaders. Like every data initiative…

Read more

4 years ago · 7 likes · Vin Vashishta

High ROI Data Science

The Thin Line Between DS And BS

The motto of most data science teams should be, 'You can't be doing it wrong if nobody knows what you're doing.' Pull back the covers on most models, and you'll find deeply flawed methods. But no one…

Read more

4 years ago · 16 likes · 1 comment · Vin Vashishta

You have recently highlighted the importance of data product managers for enabling high ROI data initiatives. Why should leaders start paying more attention to this role?

Vin: “First, we have to realize that C-Level doesn't understand how to monetize this. So they're doing a traditional opportunity discovery process, which does not include data opportunities. They are looking at everything from a digital lens, if we're lucky. But more than that, we're also getting more of a legacy lens brought to modern technical strategy.

So that's driver one, is the C-Level isn't doing opportunity discovery with a full picture of what's out there that's threats and opportunities. We're also not exploiting the opportunities available to us in our existing product lines. So we need to have this top down and bottom up opportunity discovery. Top down is very focused on TAM, your total market, that big, big picture. Bottom up is focused on your SAM and your SOM.

How much of that market can we capture and how much can we service? Now you're hearing efficiency projects, you're hearing new features for existing product lines. How do we extend the lifespan of existing products using data, leveraging new features? How do we get technology introduced into workflows. Once we do that, we have the opportunity to deliver data, do experiments, provide functionality to customers.

That's what needs to happen, and a data product manager does that. That's lesson one and two in my class, it's just that. Because it's so high value, as soon as you get that perception change driven at the top and the bottom. It's the beginning of data literacy, but really moving towards what's model literacy, and that's our next step.

We can't just make people at every part of the business understand, okay, data is this, but it's models. Models are different. They don't work the same as digital products. We have to start making the organization making the entire enterprise more model literate. That means they'll be able to articulate their needs better.

That happens at the C-Level explaining we have these strategic goals, here are our needs, what initiatives could meet those needs? Data product managers can help. Frontline, here are the opportunities we are seeing here are the underserved customer segments ,and even internally, here are the underserved user segments.

Data product managers are out there finding these opportunities. What's happening right now is data science teams have to do all of this. It is rare to find someone who can do that on your data science team. It is a completely different capability set and you rarely have someone who has that combination of technical and strategy who can play this role?

So we're putting people, sort of forcing people into service who are not in the right capabilities framework to do this. They're set up to fail. It's also asking a whole lot because what you have to do in order to get these early initiatives out the door doesn't scale. You have to spend so much time with customers, so much time with your front end users, so much time with your C-Level.

That as soon as it flips and you start saying, okay, we're gonna scale and we're going to handle more use cases, we're going to be doing more initiatives, you suddenly hit this massive roadblock because you don't have anyone to take on this work. The data team is asked to do their job and play a strategy role and play a product facing role and manage stakeholders, and you can't do that.

You just can't ask your data team to do that without introducing another role. So that's why data product managers are critical, and by extension that's why technical strategists are critical. That's why strategic leaders at the organizational level, running data teams are critical. Bringing in a CDO or a CDAO early is not a bad idea. Having them be able to take the entire team, just process end-to-end, growing the business around data, it's a different ecosystem and you need new roles.”

Dive deeper into this response with the following suggested reading from Vin:

High ROI Data Science

Stupid Users And The Need For Data Product Managers

People are the root cause of most complex model failures. We immediately think about the people who built the model or generated the datasets used to train the model. It's easy to forget the user. In…

Read more

4 years ago · 3 likes · 1 comment · Vin Vashishta

High ROI Data Science

AI Won’t Be Worth Much Until We Build Products For People VS Tasks

The data science field is starting to build products. When I began working on machine learning products in 2012, there were only a few examples of them to learn from. There were successful pilot proj…

Read more

4 years ago · 6 likes · Vin Vashishta

Given the changing market, what advice would you give leaders in rethinking their data strategy in a world where capital is expensive again?

Vin: “It has to be a sustainable business model. We've forgotten what that looks like. It needs to focus on-- I say this a lot-- great data products don't start with technology, great data products start with users and customers. It starts with their needs. You find an unmet or underserved customer segment, and that's where you start data products and you work your way back, find use cases and needs that cannot be met any other way.

That's what changes, is we have to do this in a more rigorous way. We have to create frameworks or it's chaos. We have to introduce some order to the process of selecting opportunities, defining initiatives, measuring success, prioritization frameworks are critical. We need self-service tools because data teams shouldn't be handling reporting requests.

We got infinite technology in the universe and we're still acting like we're driving around in a Ford POS. I mean, just it's so backwards right now. So we need to approach this from a completely different perspective, and that is monetization cash first. Not growth at all costs. Growth at all costs should be an organic thing.

Great products are going to grow organically as long as the company understands how to do basic sales and marketing organically. Look at OpenAI. That's the blueprint for how you get communities built and ecosystems built around products. Microsoft is teaching that masterclass right now on how to monetize models where people pay for inference, not for some feature you're building on top of it. They are paying for raw inference.

That's why OpenAI is such a valuable company right now is they're the first ones who have ever had a viable AI or machine learning as a service business model. It's the first one we've ever seen, which is viable. It's not BS. This isn't vaporware. It's actually able to power functionality and features people will pay for.

Once we get about six months to a year of track record, this will shift perception. Because it's real, and that's when CEOs start getting interested is when they start seeing cash, strategy comes forward. This is going to be the shift in thinking: how do we actually make money?

Now the CEOs see the technology is viable. Why? Because things are actually being delivered that are products that are working. They can see it work. They can put their hands on it. They can use DALL-E-2. They can use Stable Diffusion. They can use ChatGPT. They can use all this stuff. And for the first time, they know they're using a model supported product.

So CEOs are going to get very, very interested in anyone who can explain how to make this business model work. That's the opportunity for data scientists who are looking for a different place to go in their career. If you have that business side, you can be a strategic leader, you can be a data product manager, you can be a technical strategist.

You can go into one of those roles and help the business redefine how it functions. Redefining the business model to include a technology model, helps the company sort of frame this journey, this continuous transformation that they'll be going through. But if we don't connect it to value, there's no way to justify the cost.

That's the shift in thinking we have to create sustainable, viable business models. These have to pay for themselves incrementally. We can't say growth at all costs, it'll sort of maybe come together later. We have to have a definitive timeline for returns. We have to talk about break even. Those words have to come out of our vocabulary.

When will we break even? When will this initiative be profitable? How do we ensure that there is a high likelihood that we will deliver. And that we will get the returns that we're expecting. All of these are new concepts, strategies coming forward, monetization, customer needs. All of that is coming forward. Those are all the main changes that we're going to see coming in the next 12 months.

Some data teams will successfully navigate the new environment. They will thrive. You'll see investment in them, even in companies where revenues are declining. You'll see growing data teams because the monetization is there and it's proven. In other businesses, you're going to see the data team commoditized.

They will be smaller, they will be focused on some pretty rudimentary use cases, not going to be fun to work at those companies, and the demand for some roles will suffer significantly. Some data scientists are going to struggle to find a job because they're slowly realizing that what they've been doing for the last five years, seven years isn't really data science.

It's more what self-service tools now are starting to handle. Data scientist is gonna be challenged. This will be a challenging, but really opportunity filled time. So I don't mean to sound gloom and doom. This is a great time to be a data scientist. We just have to change a few things.”

Dive deeper into this response with the following suggested reading from Vin:

High ROI Data Science

What Does Data And AI Strategy Create, Add, And Solve For Businesses?

Strategy is the evaluation of tradeoffs, so the inescapable question is, should we have a data and AI strategy? This question came from the audience during a recent discussion with Andreas Welsch, VP…

Read more

4 years ago · Vin Vashishta

Person Profile:

Vin Vashishta is the Founder & Technical Strategy Advisor at V Squared. Feel free to connect with him on LinkedIn, Twitter, and his Substack to learn more about his work.

P.S. Are you looking for a community of data professionals actively growing their strategy skills? Then please check out my friend, Nicole Janeway Bills’s, community Data Strategy Professionals! I’ve gone to a few of her online networking events have met some awesome people that are also interested in strategy.

What are your thoughts on 5G?

You may not know this, but one of my dreams is to go into venture capital to embed myself further into my love of startups. Last year, I had the opportunity to delve deeper into my passion for venture capital by interviewing for a senior associate role at a venture firm. I was tasked with creating an investor memo on an early-stage startup, and it was during this process that I was exposed to the potential of streaming data and 5G network infrastructure. My research showed that the implementation of 5G is analogous to the impact broadband had on the early internet through increased speeds. It will have a similar effect on how we handle and utilize data. In short, 5G enables the mass adoption of streaming data workflows and using data-intensive workflows on our edge devices— fundamentally changing how everyday users consume data.

Announcement:

Real a quick shoutout to the Data Teams Summit conference, which will host a LIVE session of the Scaling DataOps Newsletter on January 25th, 2023! For this session, I will ask data leaders at various levels how they handle their data infrastructure when they face scaling issues. I can’t wait to hear my guest’s insights!

You can register for this free virtual event here!

Hear from Zack Hendlin, Founder & CEO of Zing Data

I am beyond excited for you all to hear the insights from Zack Hendlin! We met via LinkedIn, and I knew he would be the perfect person to go into more detail regarding the intersection of streaming data and 5G. He is the Founder and CEO of Zing Data, which focuses on enabling business intelligence on edge devices such as mobile phones. This shift of “everyday users consuming data differently” is already here, and Zack and his team are building it. Zack shares some use cases from the field to give you a glimpse of what this future holds.

The core product of your company is enabling individuals to utilize intense data workflows via mobile devices. How will this new user behavior impact both opportunities and challenges for data practitioners?

Zack: “Yeah. I think the huge opportunity is that way more people at a company are now able to use data. So if you think about folks who are literally driving trucks or working in a retail chain, we don't think of those folks as traditionally like users of BI tools. But this now lets them use that, and we, in signup behavior, see that they actually want to use it.

We had a big company that grows berries, and the person who signed up wasn't like the IT manager, it was someone who works in the field picking berries. Same with a big company that runs events. And so the opportunity is way more people can be getting value from data. We refer to it as the other 90% of the company who's not on a BI team, who's maybe not a PM or an engineer, now can use this.

The challenge that creates is if you're building data pipelines or doing data engineering, you now need to sort of think about a wider set of uses, right? So maybe it's not just, "Hey, am I gonna hit the quarterly numbers? How am I tracking relative to a goal line? Or what's my margin?" It might be, "Hey, how am I tracking on inventory, and what do I need to order more of so that I don't run out by next week?" And that's a little bit more of a kind of in-the-field use case where you want to make sure that you're creating the right aggregates and pre-processing the data in a way that makes it usable because your data user is not necessarily someone who is in R or Python. It's way more likely someone who has used Excel a little bit and who is out in the field, and you don't want them to have to parse JSON to get something useful as a result. So you need to put in a little bit more thought upfront to kind of simplify that.

I would liken it to the way that iMovie on your phone or TikTok on your phone sort of tries to make it easier to edit. You still, though, need to have a good video to edit. You could shoot video on your phone, but if it's really a lot of background noise and wobbly, even if you can edit it in a lightweight way, it's still not gonna come out great.

So I think that the onus is on data engineers and data scientists just to make sure that there are good examples people can learn from and that they're thinking about this broader set of use cases. For data users who might look a bit different from the traditional PM data scientist engineer that they've, created a table that has a hundred columns, and it's like, "Hey, go figure it out. It's all there. Go read a wiki about it." Now I think it shifts it to a little bit more of let's create some good examples that people can click into immediately, get something useful from, and then build on.”

I’ve been fascinated recently about the impact of 5G networks on data. Given that you are in the mobile space, how do you see the rise of 5G changing what’s possible for data products at scale?

Zack: “Great question. 5G in 2020 was about 3% of folks worldwide. It's much higher than that in the US, but only about 3% of people had 5G access, according to Statistica in 2020. By 2030, that's going to be 64%. So more than half of folks globally. And what that means is there's this whole new set of data creation and data consumption use cases that open up.

So by data creation, I mean IoT devices. And we actually have seen this amongst our users so far, where maybe you have an oil field, and you wanna have the devices that are going around, whether those are excavators or other pieces of equipment and know, are they up or down right now, and what are their GPS locations?

And that lets you know in real-time if things are functioning the way they're supposed to. If there are temperature readings or gas readings that are outside of tolerances, you can be firing that information every second or at least every minute. And those types of data creation use cases get opened up.

Which then lets you do really cool stuff on the consumption side. You can then set up things like real-time alerts. So instead of waiting a day or an hour to know that something is broken, you actually can know within the span of a second or less right? Cuz this data is getting created, it's getting uploaded, there are these real-time ingestion engines, Kafka, or there's a bunch of other like versions that make it a little bit more consumable, these pub-sub type models. And then you can consume it. And we built a really lightweight kind of interface where you can say, "Hey, I'm gonna tap on a graph, let me know when it drops by X percent, or goes above X or goes below Y." And you can do that on your phone and get a push notification up to every minute. And so that makes it way easier to consume all this real-time data that's coming in. And so I think the big opportunity is you're gonna go from big data sets that take a long time to create and analyze much more real-time use cases.

An example of that is we have a company that uses us in their retail chain. And what they can do now is say, "let me know every minute I have a product with inventory that drops below a hundred units that are selling really fast," and they can know that within a minute of that happening. And so it changes the whole like nature of, "Hey, I need to plan on a monthly basis."

You still may want to plan, and you still may wanna forecast, but you can respond much better in real-time. Or we had a company in the energy space, and they wanna know when there are outages. Like when your power goes down. And oftentimes, actually, 5G and wireless networks still have battery backups. So you still may have wifi wireless coverage even though 5G coverage, even though the power grid is down. And so they can say, "Hey, what houses are down right now?" And they can go check them in real-time based on the IoT data that's coming off of sensors. And that opens up these whole new use cases that are really exciting.

And then I always think about what does that actually mean? Like why do we care? Why deal with all this? There's more work that goes into streaming all this stuff in real-time, and ultimately it comes down to you can be radically more efficient if you're making a decision before something goes wrong or before you run out of inventory or before your customer queue length builds up, and customers are really upset about something, you actually can address problems before they become big problems.

And basically, you can make stuff just way more efficient. I think 5G is this really interesting intersection of devices now being good enough to do cool stuff on 'em. If you think about data on your phone today, it's still in its nascent stages. I liken it to email 15 years ago.

And 5G is part of making email on your phone better, but so too are more powerful devices— things like autocorrect. Remember when the iPhone first came out, and there was this like "sent from your iPhone" little thing along the bottom, and attachments weren't very full-featured? It was like hard to type.

Now there are dictionaries that are personalized to each user, there's auto-correct if you type something wrong, there's in the background, it will upload attachments. There are much better ways to display it. And so email has gotten way better cuz devices have gotten better, and networks have gotten better.

And that same trend is helpful for data being more usable on its own. So I'm excited cause I think that, and then one other cool thing, which you didn't explicitly ask about, but things like DuckDB and kind of really efficient ways to use data in an offline mode. If you manage those two things together really well, you can create fast, lightweight feeling experiences, even though you may actually have a huge data store underlying it in some data warehouse that's petabytes. So like we work with Databricks, we work with Snowflake, we work with these like really big data stores if we want them to be, but to an end user, we don't want them actually to have to think about like the data size. And we wanna do all these like clever optimizations that mean when they tap something, it's fast, it's responsive, and in an offline mode, DuckDB and other stuff like that we think can even extend that so you get most of that functionality even if you're offline. And then when you're back online with devices being able to store a bunch of stuff, 5G being able to send over relatively large datasets quickly. You actually can create these pretty great experiences of querying, visualizing, interacting with data on a phone that wouldn't have been possible three or five years ago.”

Throughout your career you have led product initiatives at large scale companies, hyper-growth companies, and now you own startup. What are the unique challenges of building successful products at each level of company scale?

Zack: “Yeah. I think starting at the kind of smallest size and then working up. The most important thing is a small company. Is building something that is useful to people because you don't have huge reach yet. You don't have a big budget yet, typically. And so unless it's like substantially better than what's out there, folks are not going to know of you or trust you.

So even if they come to know of you, why should I trust you? Why should I connect my data source to you? Which is a pretty big thing. Or if you're Facebook, why should I give you my personal information? Or LinkedIn, why should I put in the upfront work to fill out my profile?

So build something that people really want is the base aim of any product. And that is particularly true as a startup because it's not like you have 10 products you're offering. It's not like you have a big sales team who can sell it. There are some exceptions for big enterprises like focus companies, but for the most part, if you're product led, it needs to be understandable to someone and valuable to them.

So build something people want is universally true and probably the most important thing for a startup. And then, as you're getting people to use it, especially if there's some social or network component to it, help people understand how it's better together, better with their friends, better with their team. So we built at mentions, we built like shared questions, all that type of stuff that then lets you pull in other folks from your team. So you're like, "Hey, I have this cool analysis. I see that most of our customer support time is spent on. I now want to tag my colleague who works on customer support and have her go create more like billing FAQs so that the time spent on that goes down." And so build ways for people to share it and get more value as the network grows.

And if you look at most big companies like LinkedIn, Figma, and Facebook, that are in like the collaborative space. They all have flavors of this, make it work well in single-player mode. Facebook worked well in single-player mode cuz you could see people's pictures. Figma worked great in single-player mode, but then when you added friends, colleagues, and teammates, it became way more valuable.

So small company, build something people want. As you start growing figure out ways you can make it more valuable as there are more team members there or more friends there. And then when you get to like really big, like Facebook, LinkedIn scale, then there's actually a lot more considerations around like, how does this fit into a product experience that makes sense.

As an example, Facebook people initially went to share what was happening in their lives and stay in touch with friends. Now, over time, as the networks have become really large and maybe folks who knew in high school or college, and you get further from that, that network becomes less call it close.

And so the things that you might have shared when it was 300 people who you were really close to are different from what you might share when like, your uncle and your aunt and your mom are all there. And so what you offer needs to adapt. So things like a marketplace kind of start to make more sense. But there's really they've struggled with that close intimate sharing that used to be there. And so for bigger companies, I think a lot of it boils down to how does this fit into what people use us for. If LinkedIn says, "Hey, we're going to offer viral video," like it hasn't quite done it nearly as well, I don't think, as someone who said, like TikTok, "I wanna make this easy and fun and lightweight." So as you're bigger, you need to think a little bit more about product strategy and how all these products fit together.

And then the second thing you need to think about is, does this scale well? Is this a thing that we should be doing that makes sense, given our network, given our scale to do? A lot of things a startup might do really well, like amazing customer support, big companies struggle with as an example.

So always build something people want. Start out making your product amazing. Over time make it amazing in multiplayer mode. And then as you get bigger and bigger, make sure it fits in as a suite of products or a set of use cases that really make sense when people think about your company.”

Person Profile:

Zack Hendlin is the Founder & CEO of Zing Data. Feel free to connect with him on LinkedIn to learn more about his work.

What are others saying in the DataOps space?

Zing's CEO on the Monday Morning Data Chat with Ternary Data | Zing Data | Mobile-first business intelligence

• What: Interview Zack did with Monday Morning Data Chat and Ternary Data on their podcast to talk about why the future of data is increasingly mobile.

• Why: Three questions are not enough, and you want to hear a more in-depth interview with Zack.

• Who: You are looking to understand further how the ways in which data is consumed are changing.

From dial-up to 5G: a complete guide to logging on to the internet

• What: A overview of the history of the internet and its mass adoption.

• Why: To gain the historical context of why 5G is so impactful.

• Who: It’s not enough to hear future predictions, you want to learn the history to see if it will repeat itself.

The 2022 Stream Processing Market Update Report by Bloor Research

• What: A market report of the streaming data space providing a high-level overview. Especially pay attention to the “Market Trends” section as this helped me connect the dots to the impact of 5G.

• Why: The report provides a great high-level overview of the factors driving streaming data adoption and the current players in the space.

• Who: You are trying to understand the opportunity streaming data has in the market.

What are your thoughts on AgTech?

One of the largest data revolutions is hiding in plain sight amongst the produce within grocery stores— AgTech is quickly adopting data best practices that we can all learn from. According to this IDC article, the "…average farmer generates 500,000 data points every day…" ranging from telemetry to satellite data. Yet, we all know "mo' data mo' problems" is a pillar of the pain we experience in the data industry, given data's affinity for entropy. All of which makes AgTech one of the most interesting areas to explore for DataOps given the space is 1) in the midst of digital transformation, 2) there is a tremendous amount of rich data from disparate sources, and 3) data quality is a major pain point holding back impact. I hope you enjoy learning more about this exciting space from my guest!

Announcement:

Real a quick shoutout to the Data Teams Summit conference, which will host a LIVE session of the Scaling DataOps Newsletter on January 25th, 2023! For this session, I will ask data leaders at various levels how they handle their data infrastructure when they face scaling issues. I can’t wait to hear my guest’s insights!

You can register for this free virtual event here!

Hear from Ahraz Husain, VP of Data at Growers Edge:

One of my selfish reasons for starting this newsletter is to have an excuse to talk to interesting people in data, especially within industries I have minimal experience. I met Ahraz at the Scale AI TransformX conference a few months ago, and during our brief chat, I was so excited to learn about the data problems he faced in agriculture. So I had to get him on the newsletter so you all could learn more about this space too. Even if you are not in agriculture, we can learn so much from this industry, especially in handling the anomalies caused by the pandemic.

When people hear about AI, many first think of self-driving cars or recommendation engines behind social media, yet agriculture has a wide array of AI use cases that can strengthen our food systems. For the folks unfamiliar, what are the unique data challenges you face within this domain?

Ahraz: “Before I jump into the challenges, let me give you a few examples of the AI users in agriculture. Apart from, you know, the usuals: transportation, pricing, supply, demand, robo-workers that's an emerging space as well. But there are a few other AI use cases, probably a lot more dominant than these or very specific to ag.

You have agronomic prescriptions. What seed should I plant, where, how much, and what chemicals should I apply? So there isn't runoff, yet the crops can have the right amount of nutrients they require. Do I need fungicide for this tree or not? Then you have the good old question of crop monitoring, this has been probably one of the first sci-fi ideas, they're using satellites to monitor crop progress, how's corn doing, how are the trees doing, and such. So crop monitoring has been really old, but monitoring isn't a big challenge.

The challenge is identifying causes that are more AI-driven. Why did yield fall in this geography or this part of my field? Then obviously, you have a good old production predict forecasting, how much yield can I expect? And again, when we think of production it's at a macro level. US-wide, how many grapes are we going to get this fall or whatever?

Or on the flip side, there are challenges even within field levels. Here's a small field, 40 acres, what can I expect out of this field? If I'm a farmer, that is what I care more about, for obvious reasons.

Then you have things around leak detection. We apply fertilizers all of a sudden here in this river, there are a lot more new nutrients, nitrogen, phosphates, or whatever shouldn't be there. Can we use AI to help track it all the way back to where it came from? Then you have regulatory compliance, EPA, and many other compliance challenges. And then last but not least, this whole emerging field of carbon, carbon offsets in ag, is being driven by AI.

So these are all examples of where AI is being used, and when it comes to challenges, they're abundant as with everything, but my biggest thing is... there is a lot of data Mark. More data than anybody could imagine.

So just for context, let's talk about corn fields cause I'm here in Iowa, and it's always fun to talk about corn around here. So you've got seed, right? An acre of land can have 32,000 kernels of seed, and today's planters can really track them down. So you can really have a GPS code, which is tied to each seed planted. Now you end up with the right chemicals. So you have information on those. So just, let's think about just simply 32,000 data points per acre. Now there are 19 million acres of corn in the US it's a huge amount of data is created.

The second biggest challenge is the data is noisy. Cause up till now, and this has been a big challenge, calibration was an issue, right? So I harvested corn, for instance, or soybean, and I forgot to change the crop selection. Little things like that, all of which now AI is helping too because now we can auto-detect crops and stuff with harvesters. Then obviously, noise. Talk about noise, satellite imagery, we talked about remote sensing to detect crop performance, but cloud, now they're using AI to go through clouds spread again. There are new sensors and new technologies that can do that.

The single biggest thing, much of ag, is driven by geospatial computation. And GIS technologies didn't really go far over the last few decades, but over the last two or three years, maybe they've come a long way. Now we can run geospatial systems in Spark, but I wouldn't even call them production grade even today in Spark, but things are getting there.

Then there are challenges around access. Very few companies or organizations actually create the data. So think John Deere. Think Caterpillar, big equipment companies, really up till now, they've been exchanging it, but now they wanna get into the data processing game. It makes it challenging for startups and other innovative companies to come and play.

Then ownership, there's this big question, who owns the data now? Is it the companies that collected the equipment that you used? Is it the farm manager? Is it the owner of the farm? Is it the landlord? Is it the companies like us that process data for analytics, among others?

And then obviously security is a big deal, we've had cyber tax and all those challenges even in ag. So again, you are never short of problems to solve.”

A major aspect of data products in agriculture is forecasting produce demand to help growers understand their key business metrics— which was heavily impacted by covid supply chain changes. How can data teams navigate handling such a huge anomaly in historical data in providing accurate forecasts?

Ahraz: "What a great question. I'm gonna take an agriculture spin on this a little, just thinking about forecasting produced demand, right? It's a function of your expected production, export demands, and then ending stocks. These three are the major economic factors that go into assessing and establishing what your demand might look like now, apart from obviously great different quality of different producers or dairy products have different demand curves. And then geography plays a big factor, too, right? You can't ship corn out of Utah, but you can out of Iowa. Cause we have a lot of shipping routes through the rivers here. So those are factors. Obviously, we've always had challenges with determining supply, and I know your question's more about demand, but let me pick on supplier a little, too, just as an example.

Weather is the single biggest influence on supply; weather you can never predict. You can come up with the best estimates, but best of luck trying to get an accurate number.

I'll pick on an example from 2012, there were widespread droughts across the Midwest US, and I think we lost roughly 25, 30% of all expected yields by the end of the season. So that's a huge number. It was a brutal year for ag, at least row crops. And then you have diseases, you have things like insects, and product failures, all of which can throw your supply off completely.

But talking about demand, right? The question you specifically had. One of the biggest demand signals is through economic reports because that really tells you where to go, right?

So the USDA puts out a lot of these things picking specifically on covid, right? USDA put out an estimate that there would be a high supply and strong exports of milk. I'm talking about March 2020, and they said you could expect 25 cents for raw milk. But by the time April hit a month later, the price was cut down by 25-30% in a span of 30 days.

And I remember reading stories about milk producers, dairy farmers just throwing milk away because they didn't want the prices to go down to unsustainable levels, and so on. But going back to all these anomalies and data that we see and that our team sees, right? Some things that I have seen us do in ag specifically have been around: literally just remove the seasons that are anomaly years because you're trying to make predictions of the forecast, anomalies that can be tricky and anomalies can happen due to so many different reasons. So simple way, just remove those bad seasons. So again, from a data standpoint, not an ideal way, but from an agronomic or economic standpoint, you might do that sometimes.

Obviously, you can substitute your outliers in the data, but now we're talking about a lot of data, a lot of outliers. So that by itself is heavy compute, heavy processing, right? I've seen some models leverage pre-covid expectations. So come March, here's what we anticipated, let's use that as an actual, right? Because within 2-3% is always where we ended up end of year and so on. You can model demand, right? What would have happened if Covid wasn't there? But again, we are now getting into synthetic data, which can create its own nuances.

Then if you are really smart, you can do event-aware forecasting, which again is a different ballgame because we are talking about an unlikely event. What were the odds that covid would hit? And through every model of pretty much, right? And even economists didn't know what to expect. Expecting models to know would be a very tricky thing.

And then finally, but not least, depending on your modeling framework and in ag in agriculture and food, usually the predictions are made on so much as time series as they are, like pre-season or post, right after harvest is when people try to model and forecast demand. You can always have economic demand signals, macro signals are playing, right?

Really, depends on the modeling approach, depending on the problem you're trying to solve. Sometimes as easy as removing the data and your input. But the best thing you can do is always have an eye on economics, which is a major driver in ag, at least. And the second cool thing you can do is build your models to be agile. And when I say agile, I mean to be able to train them quickly as situations change. And to be able to make quick predictions. If you're trying to make a model that does things once, you train it once a year or once a season and make predictions once a season while you're off for a fairly rocky time, and things don't go to plan. So really, options are limitless, and that's the challenge, right? That's the beauty of our field.”

In a span of four years, you have gone from a senior IC role, to a technical lead position, and now you serve as a VP. How has your approach to solving data challenges evolved with your career progression?

Ahraz: “Questions like this make you retrospect and really think about how you evolved. Cause oftentimes, you simply focus on where you are versus how you got there. So it's a great question to ponder upon certain things that have happened.

To begin with, a really important thing for me has been learning from leaders I look up to. Find a few leaders you love. One thing I've known forever is don't follow them, but learn from them. Always challenge, and ask questions. So really, that's been the key for me, at least.

Now to answer your question very directly, there have been three fundamentals that I've evolved over time and have stayed really persistent for me. One is perseverance. At least in the data domains, failures are a lot more dominant than success stories; you try 80 models, and 19 work out. And so perseverance is the single most important thing that I've truly learned and I have benefited from.

Second is flexibility, so what that means is knowing when to back off, and you know that good old failing early concept. When it comes to modeling, the sooner you realize something might not work, the better off you are. It's better to know what are early indicators of failure are and really know when to change course.

And then finally, but not least, this is what Silicon Valley has been talking about for decades, is that first principles thinking. It's not about the solution but really the problem at hand, how this problem should be solved, and so on. So I guess those are things that have really stayed consistent for me. Obviously, what they mean has evolved and changed in my own role and what I get to do.

I guess my first role, it was everything was very ad hoc. Anything that I was asked to do, I would just change gears and do that, it moved over to a place where working bigger teams and leading teams, it got to a place where I realized real quick that doesn't work the best either. So somewhere in the middle, the flexibility to take different tracks with different projects that you're building has truly been the key. And I can not emphasize enough on how important that is and let alone for a startup, the one I'm at right now. It's important to realize the same process, project life cycle doesn't work for everything, not in our domain of data analytics. My own team today follows at least three different life cycle processes for different types of projects we do, you have the A-type analysis projects, you have the B-type, the build process, and then you have kind of a mix of both, and so on.

So I think it's really important to understand that as well. And I used to joke, so one of my titles, before I became a lead, was a solutions architect. And I used to joke once I became a manager, "I'm more of a problems architect." So it's not about the solutions, it's more about finding the right problems that are worth solving. And I think that, again, is really important, as important as the solution itself.

I remember one of my ex-bosses saying something, and he quoted, " the most successful people are those that think about implications." And the more you can be obsessed with the implications of your choices and decisions through the modeling process, through selections of projects and products, and whatever else it is that you're doing, the more likely you're to succeed. So don't get lost in the problem. Always focus on the implication. I guess that's been a big mantra of mine thus far.”

Person Profile:

Ahraz Husain is the VP of Data at Growers Edge. Feel free to connect with him on LinkedIn to learn more about their work.

What are others saying in the DataOps space?

How COVID-19 is disrupting data analytics strategies | MIT Sloan

• What: An article highlighting how the pandemic is disrupting data workflows and what firms are doing to account for it.

• Why: I can't think of any other event in recent history that changed everyone's behaviors both at a global scale and at the same time; thus, what do we do with this anomaly presented in our data?

• Who: You are a data leader still navigating the pandemic's havoc on your data quality and production models.

The Importance of Data Quality in the Agri-Food Sector

• What: A high-level overview of the data quality challenges faced in agriculture.

• Why: I like to start as high level as possible whenever I am learning within a new space.

• Who: You are trying to understand the pain points experienced in agriculture data.

Better data, higher impact: improving agricultural data systems for societal change

• What: An in-depth review of the data challenges faced within agriculture, the opportunities, and roadblocks.

• Why: Seeing similar data challenges faced by industries outside my own is an excellent reminder of how hard working with data is and how early we are in getting the most out of data.

• Who: You are potentially looking to solve data problems in the agriculture space and want to identify opportunities