Featured image for “GenAI in the Enterprise w/ Raj Shah, ML Engineer at Snowflake”

GenAI in the Enterprise w/ Raj Shah, ML Engineer at Snowflake

Today’s episode of GenAI in the Enterprise features Raj Shah, a Machine Learning Engineer at Snowflake.

Raj built his career talking to various companies and their data teams, showing them how #GenAI (and which tools) could help them reach their goals. Now at Snowflake, he gets to share Snowflake’s suite of #GenerativeAI tools, teaching them how to use them to take their business to the next level. Zach and Raj dive even deeper into Snowflake, how it works, and why companies may want to take advantage.

View This Episode On:

About The Generative AI In The Enterprise Series:

Welcome to Keyhole Software’s first-ever Podcast Series, Generative AI in the Enterprise. Chief Architect, Zach Gardner, talks with industry leaders, founders, tech evangelists, and GenAI specialists to find out how they utilize Generative AI in their businesses.

And we’re not talking about the surface-level stuff! We dive into how these bleeding-edge revolutionists use GenAI to increase revenue and decrease operational costs. You’ll learn how they have woven GenAI into the very fabric of their business to push themselves to new limits, beating out competition and exceeding expectations.

See All Episodes

Partial Generative AI In The Enterprise Episode Transcript

Note: this transcript section was created using generative AI tools like YouTube automated transcripts and ChatGPT. There may be typos, slight content changes, or character limits for brevity!


Zach Gardner: Ladies and gentlemen, welcome to the future. I’m Zach Gardner, the Chief Architect at Keyhole Software. A few months ago, I went off on a little journey—maybe a sojourn, if you are so inclined. I wanted to learn more about generative AI and machine learning. I just felt like I really didn’t know enough, especially since every conference I attend, someone is talking about it. Every time I go out for coffee with some of my programmer buddies, somebody’s talking about it. And really, you can get us going if you go out for beers—everyone is talking about it.

So, I scoured the four corners of the worldwide web and found a really cool group of people who were willing to share their insights. They gave me a much broader and better perspective than I could have gotten if I had gone on this journey alone. Today with me is my good friend Raj, who was at Hugging Face, which has the best logo in the business, by the way. Now he is an ML engineer at Snowflake. Raj, how’s it going, dude?

Raj Shah: It’s going well. Excited to be here today.

Zach Gardner: Very cool. And for our more litigious audience members, the views and opinions expressed in this program are those of the participants and do not reflect their employers, any trade, or any magazine subscriptions they may have—if that is even still a thing.

So, to get us started off, Raj, why don’t you give the audience a little bit of an introduction to yourself? How did you get into technology, and what has your career journey been like so far?

Raj Shah: Oh, my career journey? It’s been all over the place—back and forth. But I’ll start with where I’m at now and give you a little bit of how I got there. I recently joined Snowflake, where I focus on talking to enterprise customers about generative AI, AI, and how they can succeed and use it. Of course, I work for Snowflake, so I talk about how Snowflake’s stack can do it.

I’ve been in the space of AI and machine learning for about the last 10 years. I started off the first few years working as a data scientist at places like State Farm and Caterpillar, so I have a sense of what it’s like to work inside large companies and try to build a project. You go through all the headaches—data governance, not being able to get pseudo-access on a box to install some software. I know those pains.

Then I worked for those companies, but you only get to do a little bit because you’re at one company working on two or three projects. So, I moved over to the world of AI and machine learning tools, where I worked for different companies like Snorkel, DataRobot, and Hugging Face. I talked to enterprises about the different tools they can use for AI and machine learning. For me personally, I liked it because I got to talk to 10 or 20 companies in a given week. I could talk to insurance in the morning and manufacturing in the afternoon. I got to meet lots of different data science teams and see a lot of projects. That’s what gets me pumped. I really like the world of machine learning.

Of course, I’ve spent the last two years at Hugging Face, where generative AI is the ground center for all that stuff.

Zach Gardner: Yeah, it seems like something new comes out every day. Was it last night when the Nvidia CEO came out with the new chip?

Raj Shah: I know, it’s so hard to keep track of this stuff. It’s really an amazing time, and it won’t last forever. We’re really at this inflection point where there are so many announcements on a regular basis. These announcements, if you’re in the generative AI space, actually make a difference. There’s concrete improvement. We’ve seen these fluxes before in different technologies. I tell people it won’t last forever—it’ll get stable and boring after a while. But if you like change and seeing something radically mind-blowing on a regular basis, this is the space for you.

Selfishly, as a podcast host, it’s a great topic. Couldn’t have picked a better one if I tried.

Zach Gardner: So, talk to me a little more about what Snowflake is. It’s really big in healthcare, so I have a pretty good grasp of it, but what are Snowflake’s offerings? And maybe more specifically, your role. It sounds like so much fun to talk to different industries like insurance and construction, addressing their specific needs and the things they really need to be thinking about.

Raj Shah: I’ll give my own version of the Snowflake story. I have no idea how true it is, but I remember being in this space 10 years ago when everyone had to put their data someplace, typically in a database. Database admins were the power people inside IT organizations. It was one of the most complex, demanding jobs to keep databases up and running. Every company had a group of people doing it, with dedicated hardware and software from companies like Oracle.

Snowflake emerged with the growth of the cloud. You can think of AWS and all those cloud services with the idea of creating one enormous database that everyone shares. Snowflake has a huge cloud database with a lot of security and governance to ensure no one’s data commingles with others. You plug into it, access your data, do your engineering—all that stuff—without needing as much skill. The skill level for data administrators has gone down, making it much easier for folks to use. That’s why so many organizations have their data inside Snowflake. Snowflake had one of the biggest IPO offerings in history and did so well because everyone is putting their data there.

In the last couple of years, the big use case for data has been machine learning. Traditional analytics look back historically at the data to see trends, but with machine learning and AI, you want to make better decisions going forward, do forecasting, and identify which customers are likely to do something. Snowflake has added machine learning tools so people can do all that while staying right next to their data. They don’t have to move their data to another place, which could put them outside of the security and governance boundary.

For someone like me, who has worked in enterprises, I know how much they care about their data and its governance. When you’re just working on your laptop or a project at university, it’s not a big deal. But for enterprises, the data is what sets one company apart from another. They heavily protect it, so it was natural for me to go to a place where data is foremost. Good data is essential for AI and machine learning projects.

Zach Gardner: Yeah, having massive amounts of data is so important. You could have the coolest data in the world, but if you only have three records, you’re not going to train something very meaningful off of that. You mentioned data governance, data privacy, and data security. I’m curious, for some company out there, maybe they create bagels because I don’t know why, but I think I’ve gone to New York a bit too much—every example I think of revolves around bagels. But like, if they wanted to create a large language model that would produce the ideal recipe for bagels, what are some of the things they should be thinking about? Are there any tools that you would recommend, either for or against? Not throwing shade on anyone, just sharing ideas on what people could use.

Raj Shah: As a starting point, the bagel company should probably do things like build a churn model to figure out who their customers are and build forecasting models to see how much they’re going to sell. But suppose they want to dip into the world of generative AI and use that for building recipes. When we think about generative AI and these large language models, they have trillions of tokens. If you sat and read a book for something like 10,000 years continuously, you would still only have a small portion of the amount of knowledge that’s inside the latest large language models.

The amount of knowledge these models have is enormous, and when we use them, we’re retrieving and pulling that information out, trying to put it into a sensible way. But here’s one of the catches: these generative models are trying to give us an answer that we like. They’re trying to predict an answer to our question. What they provide isn’t always accurate or truthful—it’s their best guess. So, when you’re asking for a recipe, it can create one, but that recipe might not have anything to do with the physics of baking and might be entirely garbage.

This is where it’s good to have a human in the loop because working with the LLM can then probably generate an even better recipe.

Zach Gardner: Yeah, I think back to last month when OpenAI released Sora. One of the first questions I had was, “This is cool; we can generate videos. But how realistic are these videos?” I don’t know if you know this, Raj, but if I drop this pencil, it’s going at 9.8 m/s². There is physics in the world, and no amount of video training is really going to capture how thermodynamics or fluid dynamics work. It was amazing that they built an integration with the Unity physics engine because the videos that come out of that thing are just nuts.

Raj Shah: Absolutely. The integration with the Unity physics engine was brilliant. And something you mentioned reminded me of a podcast I was listening to on my way to work this morning. The head of AI at Google was discussing the current limitations with LLMs, specifically the distinction between fabrication, hallucination, and outright lies. Our brains sometimes fill in gaps in our memories, creating what might be considered hallucinations. Large language models have a bit of probabilistic randomness built into their answers, so they always provide different outputs.

Hallucinations come up in almost every conversation I have about this topic. The first company to solve hallucinations will be massively successful. I’m curious if you have any recommendations for listeners on managing hallucinations and ensuring a human is always there to grade the answers and keep things on track.

Raj Shah: Hallucinations are intrinsic to these models, and it’s hard for casual users to grasp this because the models appear confident. The results they provide look realistic, but the substance might be wrong. It’s going to take humans a while to understand that just because you see an output from a large language model doesn’t mean it’s accurate. There was a notable case in New York where a lawyer used ChatGPT to finish a brief. The AI’s citations were fabricated, but they looked logical and authoritative.

Most people right now reduce hallucinations by limiting the use of the models. One of the popular methods is retrieval-augmented generation (RAG), where the model is passed source information to get the answer from. This method reduces hallucinations by ensuring the model has access to accurate information.

Zach Gardner: It’s funny—I was playing around with an early version of ChatGPT. I gave it a dataset and asked it to run a random forest classification. Within two seconds, it gave an accuracy of 98.65%. Super authoritative, like it knows everything. Is there anything we can do fundamentally to make these models admit when they don’t know the answer? Is it a user training thing, or is it just how the technology is?

Raj Shah: This was one of the remarkable things about ChatGPT. The personality it had when answering questions felt very different from previous chatbots and large language models. It really gave that anthropomorphic, human feeling, making users feel like they’re talking to a buddy. This perception can be controlled. We can train models to be humble, to admit they don’t know, but there’s a balance to strike. The models are designed to be widely used, and part of that design includes being persuasive and human-like.

Zach Gardner: I was talking to the inventor of the App Store in a previous videocast. He mentioned concerns about dating sites using generative AI. Someone could understand human psychology and manipulate people, but with generative AI, the barrier to entry is much lower. It’s not a new problem, but it’s a more prolific one.

Raj Shah: Absolutely. We see this with generative voice technology now. Scams involve someone calling your parents in your voice, claiming you’re in trouble. They know the emotional buttons to push. This is just one example, and such scams are likely to grow.

Zach Gardner: Right. I think the Biden administration was pushing for safeguards against consumer fraud. Generative AI is moving at a rapid pace, miles per second, while policies and regulations try to keep up. It’s a game of cat and mouse.

Before I let you go, I have to ask—AGI: will it happen in the next year, the next five years? What do you think?

Raj Shah: I’m very skeptical. AGI is way far away. We’ve been manipulated into thinking these models are self-aware. They’re really just retrieving and pulling knowledge from vast amounts of data. I think AGI is much further off.

Zach Gardner: Agreed. I subscribe to the theory of quantum consciousness, and I don’t think we have enough quantum particles to achieve consciousness. But that’s a podcast for another day. Raj, it’s been an absolute pleasure having you on. Thank you so much for coming.

Raj Shah: Thanks, this has been fun.

Zach Gardner: And ladies and gentlemen, we’ll catch you in the future.