How I built an entire data platform by myself that thousands of data engineers use in one year

Thank you, everyone. So, yes, this is, I think, the fifth time I've done this one now. This is the final talk of the whole tour. So, no pressure, right? Just a little more introduction about myself.

I've been doing data engineering since 2013. Just to give you a little more background about myself, I've been involved in data engineering since the days when everyone was using Java MapReduce, Hive, Hadoop, and Uzi—technologies that many of us have moved on from. Anyway, I'm going to discuss how I built a data platform that thousands of engineers use. It should be pretty interesting!

We're going to start this off with a quote. There's a quote from Sam Altman from around November or December of 2022 where he said, "We are going to see one-person billion-dollar companies because of AI." Initially, when I heard this quote, my gut reaction was skepticism. I thought, "This is just NFTs with extra steps," or something along those lines. I don't know.

Now we're here. It's been, what, two years? And it's just getting crazier, right? It's not going away. I think we have all come to the conclusion now that this is not hype.

It's not a fad. There are also some other things that are not hype and not a fad. What other things are not hype and not a fad? Well, big data definitely isn't, right? I think that's what a lot of you are in here for, right?

You can earn great salaries in data engineering by processing data, uncovering the actual truth, and making informed decisions with big data. Another important aspect is social media and personal branding. I'm sure many of you have heard of Mr. Beast, right? He has 300 million subscribers!

He's the most followed person in the world, and he's four years younger than me, right? So it's crazy where the world is going. One of the things I find very interesting is the intersections of these trends. I think that the bottom circle, the intersection of big data and social media, is the most obvious one. That's where you get... You know, big data and social media is the most obvious one.

That's where you find the data engineering influencers. By the way, this is actually a relatively new trend, and it's quite an unsaturated market. If you look at all the data engineering creators across various platforms, there are only seven who have over 100,000 followers. There are about five million data engineers. So, what does this mean for us?

Let's support one creator per million. Okay, cool. Can my million give me $10? Thank you. There's a very unsaturated market there.

So that's something I identified in 2022. And then, obviously, you have the intersection of artificial intelligence and big data. That's where you get ChatGPT, and you get a lot of the content that is no longer just made up; it's actually rooted in facts. On the other side, you have spam masters.

This is actually something I've leveraged quite a bit in my business. Not to spam people, but I couldn't come up with a better title for this. You leverage your brand and artificial intelligence to send out sales emails, like cold emailing people and trying to get them on board. But really, what we're here to talk about is actually the intersection of all three: emails, cold emailing, and trying to get people on board.

Where you combine AI, big data, and social media, you can create a one-person empire. Something that caught my attention around January of 2023 made me realize something very important: I needed to quit my job. I had been working at Airbnb for about the last two years, and I recognized pretty early on that AI was going to create such an opportunity cost, right? Airbnb was paying me great money—one of the best-paying data engineering roles you can get in the entire world.

And even with that, I recognized that the opportunity cost of working at Airbnb was too high because of AI. So, what are we talking about today? We're going to discuss a couple of things. The big focus is on data expertise and what I have built, specifically the before and after architectures. We've completed four evolutions of the architecture now.

We're going to talk a lot about mistakes I made along the way, because I've made an unbelievable number of mistakes over the years. Hopefully, you guys can learn from me if you're building your own startups. Last but certainly not least, we'll discuss what the future might look like for data engineers. Here’s a quick summary of my history: I quit Airbnb in March of 2023 after realizing the potential of AI. Shortly after, I recognized that Airbnb was actually paying me pretty well, and I wanted to make some money.

So, in April, I decided to launch my beta cohort for Data Expert. I had 50 students. Let's just start off with the basics here: this is when I started my company. I'm a young, scrappy startup.

Remember, my goal here is to be a one-person business. I don't want to have a lot of other people involved. This is the first version of the architecture. The main thing I did here was post a Google Form.

I was like, "Do you want to join my boot camp?" Then I went through it, found the candidates, and hand-selected the first 50 people. I sent them invoices, and once they paid, I manually invited all of them to Discord, GitHub, Google Classroom, and all that stuff. It was painful, right? That was a challenging experience. In this first version of the boot camp, we actually used Databricks.

And essentially, what we tried to do was have the GitHub repository contain the code and data necessary for Databricks. The students then needed to take the data from GitHub and upload it into Databricks. We did data lake modeling using PostgreSQL, which is somewhat unusual. The benefits here are quite clear. I can easily say that I set up this entire architecture in about three or four hours.

One day. No server costs, right? Everything was free. Tons of built-in functionality, by the way. Google Classroom is actually pretty great.

It gets you pretty far. I was really surprised by how much was available for free; I was like, "Wow, this is pretty awesome." But obviously, there are a lot of drawbacks to this version, right? The Databricks free trial was absolutely horrendous. I'm sure some of you have tried the free trial and have noticed that there's a lot of pain and suffering involved with that.

With the 50 students that I taught, I was able to get 20 of them up and running with the data in time. Which is terrible, right? If I marketed my boot camp that way, I would be saying, "Hey, I'm able to teach 40% of my students Spark," right? That is not a good statistic. Also, the Docker setup is quite painful, right?

Because my boot camp students are a mix of analysts and software engineers, some of them really don't like Docker. I'm never sending out manual invoices ever again. Google Classroom had some limitations, and it actually had one really damning feature.

It's the download button. One of the things I realized after finishing my first boot camp was that I found my entire curriculum and videos on Pirate Bay one week after the boot camp was over. So, I was like, "Oh, maybe I shouldn't use Google Classroom. That's probably the wrong play." So, let's talk about the iterations.

So this is Version 2. This was the summer of 2023. This is the second boot camp I launched. This one was able to support three times as many students. We went from 50 to 150 students.

And I wrote some code. I wrote some code, right? We got a domain, dataengineer.io, which was too expensive. We'll talk about that later. One of the big things we added here was addressing the piracy problem, which was a significant concern for me.

And so we added gated paywalled access with encrypted streaming from Amazon CloudFront. That was the biggest win for the platform. However, I also got rid of Databricks; I didn't want Databricks anymore. We went with Tabular, actually, and I was like, "Ooh, Tabular's a really promising startup." And guess who stopped that?

Databricks. And so I set up my data lake on Tabular. I still have it on Tabular, by the way. It's great. We ended up teaching local Spark for the second version of the boot camp.

We also added Heroku Kafka for real-time streaming with Flink, which essentially became our second architecture. This architecture was great; no more manual invoices, and we were able to scale up.

I made 150 sales and really ramped it up, earning a lot of money. What I loved about this was that on June 28th, 2023, I made $100,000 in one day because I had 50 sales. That was the day I realized I would never go back to Airbnb. The truth is, I'm never going to make $100,000 in one day at Airbnb. There are also other factors like CloudFront, which helps with security and eliminates piracy problems. Additionally, having a domain brings many benefits, such as improved SEO and visibility on Google, among other things.

And branding, all those things. We also had a logo, not this logo. I slapped together a logo with Dolly, and it was actually one that was pretty good-looking. I thought, wow, I never need a designer.

But then a lot of people were saying, "Zach, your logo is not good. You need to get a new logo." Now, the new logo I have is in the corner of these slides, and I actually like the one we have now. Okay, risks, right?

So, in this version, we were still doing data lake modeling in Postgres. Many students found this unintuitive because they had to translate a lot of the SQL and data modeling concepts we were using in Postgres into Iceberg and Delta Lake. It wasn't very intuitive; students were asking, "Why wouldn't I just learn it on that?" Server costs weren't zero, but they were affordable, around $200 a month. People disliked that I moved from Databricks to local Spark, expressing that they wanted to do real data engineering in the cloud, as data engineering isn't typically done locally.

And then, you know, I was still manually adding everyone to Discord and GitHub. My assistant did not like that—150 people! We actually came across an interesting constraint with GitHub. You know that you can only invite 50 people a day to your organization on GitHub?

That actually was a problem in the second bootcamp. I was like, "GitHub, I promise I'm not a spammer or an abuser. I'm not trying to abuse your platform; I just want to have a successful bootcamp." So those are the risks for this one.

So let's check out the third version of this architecture. This is the third version that we have moved to. You'll notice some interesting things here. PostgreSQL is gone, right? I no longer teach PostgreSQL in my boot camp because we are all about data lake technology for the future.

And you'll notice we added Starburst and Trino. Those are the big additions here. I actually have Starburst and Trino to query from Next.js. One of the things that's really cool about my boot camp and the data modeling weeks is that you can do the entire data modeling weeks from your phone if you want. It's one of the most accessible boot camps.

You don't have to install anything. You can write SQL from Chrome and learn data modeling, which is really cool. Another big addition here was automatic invitations, right? It made things scale up a lot better and freed up my assistant's time quite a bit.

This version was great. The biggest win here was the data lake modeling. We were using Trino, we were using Iceberg, and we were utilizing current state-of-the-art technologies that people loved. It was a smash hit. During those weeks when I taught that, it was very, very good. Automatically adding people was amazing.

And one of the other things I recognized was that when I taught that, it was very, very good. Automatically adding people was amazing. When I tried to set up Trino here, one of the things I did was initially think, "I'm a good engineer. I'm going to use Terraform and the AWS console."

And one of the things I realized as a one-person company is that it is a very stupid thing to do. The reason for that is that I have to do everything. I don’t know if you know, but I have to handle sales, marketing, a lot of the administrative work, and the front-end tasks. The one piece of work that I definitely don’t want to do is DevOps. DevOps is not what I want to focus on. So, I thought, why not use a service for that? That was something that really became clear to me regarding time management, especially with data landing in the lake.

And that was something I really wanted to show my students, but we weren't able to pull that off. In this version of the boot camp, we focused on Kafka, Flink, and Postgres, which was really strange. However, we were still able to teach all the real-time concepts, just not that last data lake step. Additionally, in this version of the boot camp, I increased the homework assignments. There were 15 homework assignments for each student, and with about 200 students, that adds up to 3,000 homework assignments to grade. That's a lot!

Sure! Here's the reformatted text: "Happening, I was like, well, I solved that problem just by hiring people. I hired 20 of my old students to each grade 150 assignments, and that's what we did. However, that was not great, actually, because keeping those 20 people consistent, timely, and maintaining a tight feedback loop was impossible. There were just too many moving parts."

And that was something that I had to think about for the next version of the boot camp. I need to figure out how to approach it because, again, I want to have a small company with a few employees and a limited number of offerings. So let's talk about the current iteration. This is the current iteration of my boot camp.

You'll see there was a gap between V3 and V4. In this version, you'll notice there are a lot more things added because I had six months during which I wasn't working on a boot camp. This allowed me to incorporate many additional features. There are a couple of things I want to highlight, especially in the bottom left corner: remember, don’t roll things yourself—use services. That's what I want to discuss first.

You'll see that making an account and then using Clerk is the new thing. So, in March of this year, I wasted three weeks of my life because I created my own authentication system. Then, a script kiddie abused my signup form, sent out 70,000 phishing emails, and I thought my domain reputation was doomed. That was when I realized, why do I care about authentication at all? Like, why is this something I'm even worried about?

This is again a place where my "oh, I'm a technical engineer, I can do anything" attitude bit me in the ass, right? Because that's not the case. I spent three weeks migrating my authentication system to Clerk, and now I never have to think about it again, knowing that Clerk has it handled for me. Other things.

We were able to add subscriptions to this, which dropped the entry point for my bootcamp by an order of magnitude, allowing many more people to learn from me. Another very important development is that we added AWS for Spark and Spark Streaming. This finally allowed us to realize our vision of transitioning from Kafka to a streaming job to Iceberg. Additionally, you might notice an integration with ChatGPT here.

We noticed that homework grading doesn't have to be done by a human most of the time. As a result, we were able to create a system that allowed us to reduce the complexity of our homework and grading processes. This system provides several benefits. We moved from Heroku to Confluent, which is great because now our students can have their own topics. In the previous boot camps, everyone just dumped all their events into one topic, which was quite messy, as Heroku's Kafka doesn't allow for multiple topics.

You only need one, and it costs $200 a month. It's a gigantic ripoff. Don't use it. Confluence is great. You'll have many of them, and it's cheaper.

So that's awesome! Actually letting my students use Astronomer to deploy in the cloud and have capstone projects with things that are actually running in the cloud was another moment that made me think, “Wow, this is real data engineering now.” This is not just some local toy example; this is the real deal!

Automatic homework grading was amazing. In the fourth boot camp, we had 350 students, resulting in about 5,000 to 6,000 assignments. ChatGPT reduced the complexity of grading by an order of magnitude.

Because it got it right 90% of the time, and for the other 10%, I would hire someone. In my third boot camp, I had 20 graders. In my fourth boot camp, I had two, all while adding twice as many students.

So this is an example of how AI saved me tens of thousands of dollars. Instead of paying those graders, I can now pay ChatGPT, which costs about a hundred bucks compared to tens of thousands of dollars for a human to do it. There are several risks involved; one of them is that removing Docker made some students angry.

It made me recognize that my boot camp needs to have a couple of segments. There are some data engineers who come from a software engineering background and want to be more like data platform specialists. Then there are data engineers who come from an analyst background and never want to touch Docker. It's kind of a balancing act.

And that was a significant issue I recognized: inadequate access controls. When I set up Glue, I mistakenly provided everyone with the same key. One of the consequences of that was that one of my students ran a job that ended up costing around fifteen hundred dollars, which is essentially his entire tuition.

And I'm like, "Thanks, bro, thanks." One of the things I recognize there is that I need to do something similar to what I do with GitHub and Discord. When someone signs up, they get issued a new key. In AWS, you can actually set limits for people, so you can manage it on a per-student basis and automatically track their costs.

So I don't have to have that problem anymore. But when you only have one key, there's nothing you can do, right? Or you can only set some global limit that has to be pretty high. And another thing, just like with anything involving AI, you have hallucinations, right? Because when I do the automatic grading, I have it give me a pass/fail and a confidence score, indicating how confident it is about that grade.

And sometimes ChatGPT is confidently wrong. I'm sure you all have experienced that before. In those cases, one of the things we recognize is that the onus is on the student to raise the issue. It's like ChatGPT asserts, "This is definitely wrong," and the student has to come to us and say, "No, it's definitely right," and then we have to go through that process. That's something we are trying to work on.

I know that there's a way to do that with fine-tuning that can probably make those problems go away as we get more examples. So, let's talk about mistakes. A big mistake here: I made a big mistake in the second boot camp when I bought dataengineer.io. That domain was $8,000.

I actually feel grateful for this because I was debating between two domains: dataengineer.io and dataengineering.com. Dataengineering.com is $100,000, so I'm very happy that I made a mistake of one order of magnitude less. Why is this a mistake? Because dataengineer.io is not an accessible brand for people who don't want to be data engineers.

If you want to be an analytics engineer, data scientist, or machine learning engineer, there are many other roles that may be a good fit for you. That's why I intentionally switched to "data expert," because "data expert" is intentionally vague and more accessible. The main lesson here is: don't make big purchasing decisions without considering the implications for one, three, or five years in terms of where your business is headed. I bought it impulsively because I thought, "I'm the data engineering guy." Many of you probably know me as the data engineering guy.

And so that was something that I recognized as a pretty big mistake throughout this journey. Other mistakes included crying in the AWS console for days. I was struggling because I was trying to learn the art of being an entrepreneur, as I have been an entrepreneur for 18 months.

I was a big tech engineer for eight years, and I pride myself on my technical ability. So, it's like, why use a SaaS product? I can just build it, right? And that's something that, frankly, as a one-person company, is stupid.

Because you want to be able to open up your time to the highest leverage opportunities. My highest leverage opportunities are talking to you right here, making content, doing sales, and conducting boot camps. Those are the main activities, right? Writing code and building out my own projects should be minimized as much as possible. Other mistakes.

Not researching, so I just want to talk a little bit about CloudFront. Initially, I actually wrote my own streaming APIs, my own video streaming APIs. I thought I could do it better than Amazon, right? Which is, again, bold, but dumb. I tried to set up Trino with Terraform and do all the crazy DevOps stuff. And Heroku Kafka, Confluent was a choice, but I just didn't think about it. I deployed my app through Heroku; my Next.js app was deployed through Heroku, and I was like, "Oh, Heroku has Kafka? Let's just use that," right?

And I didn't even think about it. I was just like, it was the closest and easiest option. I didn't do any research at all. There were at least five other times when I made the same mistake. So the main point here is to do a little bit of research.

Look before you leap. Just think about what some other options are before you dive into code. Another point is not using AI sooner. One of the big things I recognize is that AI has changed my development processes.

It's probably changed a lot of your development processes as well. One of the big things that I recognize is that AI has changed my development processes. It's likely changed a lot of your development processes as well. One of the things I've recognized is that there's a sort of rule for me with AI: if you're trying to write something that is more than five lines of code but less than 200, that is where AI has a sweet spot.

That's where AI is most likely going to be better. If it's less than five lines of code, just write it yourself. If it's more than 200 lines, the problem is that you're probably going to run into debugging issues.

Because it's going to get something wrong once you reach a certain number of lines of code, or a certain level of complexity, AI tends to mess things up. However, that might change as AI improves. My main point is that I was still writing 30, 50, or even 70-line functions when I really should have just prompted the AI, plugged in my requirements, and let it do the work. This would significantly double or triple my development speed. Utilizing AI in this way would have allowed me to build this platform even faster.

I remember when I was building my Stripe payment gateway. That was something I developed with AI, and I didn't believe it at first. I had scoped the problem as a four-week task, but with AI, I completed it in just two days. After that, I published it for sales and wondered, "Is this going to work?" It felt like it wouldn't because I kept asking myself why it was so easy. But to my surprise, it worked!

It was great. I was like, "Wow, this is amazing. This actually works great." So definitely learn that AI is the future, and you need to be leveraging it as much as possible. Not using AI today is like not using Google.

So we're going to shift gears here a little bit and talk about the future of data engineering and AI. There are three key areas that I found to be very powerful for my workflows during my last five to six months at Airbnb. One of the things that was particularly prominent when ChatGPT was released was the discussion around how SQL is dead and Text-to-SQL is the future. However, it’s important to understand that English is an inherently ambiguous language, whereas SQL is not.

So, one is an inherently ambiguous language, and SQL is not, right? One of the things you have to realize is that text and SQL are never going to work well together. This is because they are trying to map an imprecise language onto a precise language, and it’s just not going to work.

Especially once you reach a certain level of complexity, it doesn't work, right? It might work for a five-line SQL statement with a GROUP BY, but when you start working with anything involving a Common Table Expression (CTE) or nested queries, good luck. What I have found AI to be amazing for is boilerplate. If you are familiar with your analytical patterns, such as cumulative tables and slowly changing dimensions, and you write, audit, and publish, it can provide you with a nice Airflow Directed Acyclic Graph (DAG) with plenty of boilerplate code that you just need to fill in with a few details. Another area where I find it particularly useful is in conducting data quality checks.

Because you know, and you need, there are three types of context building that you need to reduce the hallucination rate of LLMs. One is a vector database. This is where you store any sort of unstructured content, especially text content. It's very effective. The second is a graph database.

This is going to be any time you have nouns, entities, or relationships that you need to look up and bring in any other contextual information that way. Regarding relational databases, I feel that people don't give them enough credit because they may not seem fancy. However, keyword search is still excellent for large language models (LLMs). If you want to perform a keyword search to retrieve your Retrieval-Augmented Generation (RAG) documents and data, it works great. You don't always have to rely on Pinecone or constantly incorporate new, fancy tools.

You can just use Postgres. Agents. This is probably, out of everything, the area that I find to be the most exciting development for data engineering. If you look it up, if you Google "data engineer burnout," just Google that sometime. The numbers are interesting, something like 80%.

Many data engineers are experiencing burnout, and this is the number one reason. For me, when I worked at Netflix from July 2019 until March 2020, I was on call for nine months. That was a lot, and that's why I quit.

Well, I'm done. I don't want to work at this company anymore. If you all don't want to help me, then I'm done. One of the things I've learned from this process is that there are two types of issues that occur during on-call that are trivial and not worth our concern.

One issue is false positives, which can happen frequently. If you use data quality checks, you've likely encountered a false positive in your data quality check, where you think, "Oh no, that's wrong." Or it might be right within the threshold, leading you to conclude, "Okay, it's benign," and then you allow the pipeline to continue. That kind of logic and reasoning is something that LLMs will be able to handle, as it's not overly complicated.

Another one is Spark, right? Whenever Spark has the data shape change just a little bit, it can run out of memory. I'm like, "Alright, I've got to bump up the memory." Why do I have to do this at 3 a.m.? Why can't an LLM do this?

Right? And I find that this is going to give data engineers a mental health boost. It's going to help data engineers because I've always found that when I'm a data engineer, every pipeline I write is another pipeline I have to maintain. It feels like I'm slowly building a wall of my own demise, right? Because essentially, every incremental pipeline means another false positive and another out-of-memory error, right?

And I think this is one area where LLMs are going to dramatically increase our ability to maintain pipelines and our mental health. So, in conclusion, the opportunity in front of us is massive. AI is the future. You can build; you can create things that are quite amazing. And the time is now to build.

The time is now to create something truly amazing. And the time is now to build. Thank you, everyone.