We helped write the sequel to "Cracking the Coding Interview". Read 9 chapters for free

System Design: E-Commerce Inventory & Order Processing

Watch someone solve the amazon prime day order processing system problem in an interview with a FAANG engineer and see the feedback their interviewer left them. Explore this problem and others in our library of interview replays.

Interview Summary

Problem type

Amazon Prime Day Order Processing System

Interview question

Design a system that handles over 1 million orders per minute during a peak shopping event. Each order must reserve inventory, charge the customer, and confirm the order — all while preventing overselling, avoiding duplicate charges, and maintaining real-time inventory counts visible to other buyers. The core challenge is managing high-contention inventory updates at massive global scale while keeping the user experience fast and accurate.

Interview Feedback

Feedback about Dialectic Thunderstorm (the interviewee)

Advance this person to the next round?

How were their technical skills?

3/4

How was their problem solving ability?

3/4

What about their communication ability?

2/4

You seem to have the technical skills, but have some trouble communicating them. I recommend more practice for a real interview. Here are some of the things we talked about today. Ask questions to understand the scope of the question. You will always get half a scenario; they expect you to ask questions. You want to understand which services are provided to you vs. what you are building. That will help you understand what is important. Sometimes the notification or payment service is ancillary, sometimes it's the whole question. Keep in mind that at the large scale of Amazon, even tiny chunks of workflows are more complicated than you expect. If you are asked a question about a startup, it's more likely to have a more complex workflow with fewer users. You seemed unsure about some of the decisions. If you are going to offer two options, make sure to state exactly what their technical trade-offs are, then choose one. Making decisions and then changing them if they are wrong is a good judgment skill to have. Distributed services (one service, one db, one task) are helpful at a large scale in reducing the blast radius of issues. Make sure that if you pick a technical thing (e.g., a PostgreSQL DB), you can defend your choice. There are often more states than you think initially. Once you have a workflow, you will be asked about scaling worldwide, handling instance failures (chaos engineering), and dealing with entire service shutdowns. The answer depends on how much control you have over the design of the service (yours, another team's, or third-party). You can make assumptions about what will be better for the user, but give your thought process. Your thought process is always helpful as long as you make the decision.

Feedback about Declarative Dragon (the interviewer)

Would you want to work with this person?

Yes

How excited would you be to work with them?

4/4

How good were the questions?

4/4

How helpful was your interviewer in guiding you to the solution(s)?

4/4

My interviewer could not have been more helpful. The advice was awesome, she gave me a little bit of time to thrash about and think, then jumped in and nailed what my issue was and what a good way to move forward would be. Thank you so much!

Interview Transcript

Declarative Dragon: Hello. Hello. How are you doing today?

Dialectic Thunderstorm: I'm doing very well. How are you?

Declarative Dragon: I'm good. I'm just opening up my notes. Um, so today is— it looks like you were looking for a more of a mentoring session rather than just like a straight interview.

Dialectic Thunderstorm: Yeah, I'm— what I was thinking was, you know, I wanted to somewhat mirror an interview environment, but in my opinion, I'm not super ready and this is my first one of these. So I'm really just hoping, like, I want you to challenge me and, you know, play devil's advocate like you would normally, but I also might just need a little bit of help.

Declarative Dragon: Sure. Have you— do you have a system design interview coming up?

Dialectic Thunderstorm: I do. I am interviewing with [REDACTED] with their system design interview on Wednesday.

Declarative Dragon: Okay. So do you have any— I like to ask just in case people do. Some people come with questions. If you don't have one that you'd like to practice with, I have a whole bunch that I can grab. I'm thinking this one. Okay, I have one. Would you like to just get started on an interview question and then we can kind of see how it goes?

Dialectic Thunderstorm: Yeah, I think that sounds good.

Declarative Dragon: Okay. And then I think because this is more of a coaching session and your first one, I will do the feedback kind of like during the interview as opposed to like normally I like do at the end. I think maybe for this one we'll do it in between and then for your next one maybe just do like a straight interview like it's the real thing.

Dialectic Thunderstorm: Yeah, that's my plan. I think that sounds good.

Declarative Dragon: Okay, great. All right, so we're gonna do an Amazon question. Okay. So on Prime Day, Amazon processes over a million orders per minute. Each order must automatically reserve inventory, charge the customer, and confirm the order, all while preventing overselling, which is selling more units than exist, avoiding duplicate charges, and maintaining a real-time count visible to other buyers? So basically, this is similar to many systems design questions you get where the actual action is pretty simple, right? It's a shopping cart, right? You've seen this on a million e-commerce sites. I picked this question because [REDACTED] is an e-commerce site, except, yeah, it is because you, I'm thinking about it. I've never actually used [REDACTED], but I assume it's similar to Amazon Fresh where you just like, get what's there, but you need to make sure that it is in inventory. So it always starts with like a simple-ish question, but what makes it complex is the scale, right? And that's why we— I immediately said 1 million orders per minute to get you in the mindset of a huge scale. And personally, I don't really care about the specific numbers. I know that's part of some of the like study materials, But I do like to know that you understand that 1 million is very different than like 100,000 or even 100, right? Like every time you go kind of 10x in scale, it changes the way you would do the question. Okay.

Dialectic Thunderstorm: Yeah, that makes sense.

Declarative Dragon: So does that, does that question make sense? Basically just a shopping cart for a very busy shopping day.

Dialectic Thunderstorm: It does. It pretty closely aligns with the kinds of things I've been studying for this. So I think it's perfect.

Declarative Dragon: Okay, great. Let's get started.

Dialectic Thunderstorm: Okay, so I guess I'm just going to use this plain text to put down my initial thoughts and then bring up the whiteboard once we get to the high-level design.

Declarative Dragon: Okay.

Dialectic Thunderstorm: Okay, so I— the, the first thing I want to lay out is the functional requirements. And, okay, so the main thing that we want to do is allow, we'll say customers can view items in the inventory along with, I guess, price, but more importantly availability. And then we also want to ensure that customers can reserve items. I, in other words, put them in their cart. And I think this is probably something we want to list as part of this functional requirement, but when they reserve an item, that item's availability reflects the reservation for other customers. And then That is probably kind of going to be sort of in a temporary phase where the item is reserved. So when a customer checks out, the inventory is updated to reflect it. And at this point, this is going to be more in our nonfunctional, but we want to ensure that, you know, if they have that item in their cart, they can check out, we're not overselling, and we probably want to avoid saving our inventory check for that checkout, which is why I'm thinking you check the inventory status when they go to add it to their cart, and then we have a reservation and we hold on to that until they actually do the checkout. So at that point, we've already deducted it from the inventory and they won't get any nasty surprises. I don't know, I think it sounds like the main priority of this is going to be handling that that consistency and, uh, you know, a million customers going for a similar set of items. So I think I'm going to mostly dive into those latter two points and go into that contention. Uh, but that first one is— it, it's, it's an important question, but it's probably— I don't want to say incidental to the, the real thing because we do want to show them real-time availability, but it's going to be more how we model that data how we model those transactions in the backend. Do you think that makes sense?

Declarative Dragon: Yeah. So my initial feedback is that every question you get is going to be half a question, right? If you've watched Great British Bake Off, I always make it— I use that metaphor because it's like the second challenge they do where you get half a recipe and have to do it. So the point of that is that you ask questions, and especially for a big company like Amazon, you're gonna be able to use services that already exist to take care of things that aren't the most important part of the question. So from your reading of the question, I think you got it right where you were like, the inventory is kind of what we're looking to handle here. But like, obviously there's all the parts before this and all the parts after this that are still important to the customer process, but actually aren't important to the question. And that's where you can ask like, okay, is there a service that's gonna be, providing the catalog with the prices, right? And if you ask me that question, I would say yes, that service exists. And, you know, there's also an inventory service that shows what is available now and at which warehouse, right? So just to give you some, like, framework that you can work around, because the meat of the question is always harder than you think it is. So a lot of people will do what you're doing and kind of like build out the workflow, but that's not what you want to do. You want to like really focus it on what is the hard part of the question, which is the inventory, which you got. And then the other stuff, just ask, just click confirm, really. Like, hey, is this available to me or do I need to build it out? And the answer to that question tells you what is the important part of the question. Does that make sense?

Dialectic Thunderstorm: Yeah, absolutely. Yes. Okay, so I'm going to kind of go back in time a little bit and just start, you know, I'll start with the one that you started with. Uh, in terms of showing customers real-time pricing, that will be taken care of from another service, right? Okay. And our concern is when they're browsing, showing them real-time availability.

Declarative Dragon: Our concern is showing them items that exist. So they don't need to know that there's like 42 items. That will hurt the marketing. We want them to see it as there's one item left and they better get it before someone else gets it. And that's what's important is that they have that one item and when it's gone, it's gone. But they don't know how many are gonna be left, but we wanna make sure we don't, sell the same item to two people if we're down to one of that item.

Dialectic Thunderstorm: Okay, okay, that makes sense. Um, okay, so yeah, it sounds like our real concern here is that inventory contention and managing, uh, transactions when a customer goes to reserve and check out. I'm trying to think of other questions that I need to ask to narrow down the scope. Um, do we want Are we managing any sort of analytics on ordering or is that out of scope?

Declarative Dragon: That would be out of scope for now. And one thing to add to in terms of feedback is, you know, when I name an actual company as opposed to like a general e-commerce site, you wanna think about your experience on that company's website. 'Cause you know, when they name an actual company, it's usually like they want you to go into it. And when you interview, they will often ask you to add features to their, you know, site that doesn't exist. That's a very common system design question. It's like they pick something that they're not gonna build and have you build it. So just really think about your experience on the actual reality of a site. So like in the interview, think about, you know, I've definitely spent a little bit of time on [REDACTED]'s website and make sure you are familiar with kind of all the pages a customer would get. And then for this question, think about, you know, think about your experience on Amazon. Have you ever purchased something that hasn't shown up? Which is the answer for me over like thousands of purchases, no. And so I think it's most customers and that shows that their control of the inventory is very strong.

Dialectic Thunderstorm: Okay. Okay. So yeah, framing it in what I've experienced with Amazon, I guess my next question would be, so Once the customer checks out and they have put in a purchase order for that item, can we assume that the actual dispatching of the item from a local warehouse will be taken care of elsewhere?

Declarative Dragon: Yeah, so our focus is on making sure that they get the items that they purchase and that they're not double selling or not double charging. You know, like that little small window of the Amazon process of they're selecting an item, it goes in their cart, they purchase, and then we're done. And then everything after that, we can, you know, kick off something like usually something like that. I would say I'd kick off an event or something, and then that something else will take care of the rest of the process of dispatching it.

Dialectic Thunderstorm: Okay, okay. So yeah, we've narrowed down the scope to they select the item, they add it to their cart, and they purchase it, and we want to make sure we know if, you know, they can't add it to the cart if we don't have any, and if something goes wrong with the purchase, we don't charge them twice, and we update our inventory to reflect all of that so future customers see those changes.

Declarative Dragon: Yes.

Dialectic Thunderstorm: Okay, um, I— that sounds like a good narrowing of the scope. Are you okay if I start on non-functional requirements, or is there more that we should No, I think this is good to start with.

Declarative Dragon: And remember, you can keep asking questions throughout the interview. And I kind of like it when interviewees kind of just start with the basics. I always tell people, get to the boxes as soon as possible because it shows that you understand, like, you know, build a simple system and then a more complex system and then more complex system. Like, you can keep making it more and more complex as you think through the product and you think through what it could be and ask more questions. So don't think of it as like you ask questions at the beginning and then you're not allowed anymore. It should be a back and forth. You want to treat me like I'm either like the lead engineer or the product manager who's giving you this product to build. And, you know, we are coworkers. You can ask many questions if you need.

Dialectic Thunderstorm: Okay. Can I, can I ask something related to that? One thing that I kind of worry about is I'm going to start with sort of a simple thing that, you know, assumes we can use one database and maybe like my initial design doesn't have caching yet, things like that. Or like somewhere that I definitely know we want to have a message queue, but like I don't know if I should forego that in the first iteration. And then what happens if we don't have enough time and I leave the interviewer thinking I didn't know about like, you know, database sharding, using message queues, things like that.

Declarative Dragon: Well, I think that's why I always try to get people past the questions as fast as possible so that they can just get drawing, because I think it's, I think it can be, you know, nerve-wracking to try to keep it all in your head. So I think it's important to just get drawing as soon as possible. And, you know, you do a simple thing to show that you understand the workflow, and then you're like, well, this, this would work if there was one customer, and that, you know, that this is how I would handle it. But this is not what's gonna work for Amazon scale. But here, at least I had the workflow down. I understand that. Like you would be surprised how many people don't even get the workflow down. Like they never even finish the question, let alone finish the question for scale and scope, right? So, you know, as you go, like the functional is like the workflow. I always compare that to like the business workflow. The non-functional is like, what is the scale and scope? And we can keep going into that, but like, it should be pretty quick because I kind of already told you it in the question about like, what is the scale we want to do, which is a million orders per minute, right? Like that should give you almost enough. And then the only other question I would ask is about, you know, when we think of scale and scope, it's like how many users and where are they on Planet Earth, right? Like, are we supporting just Yukan, Are we supporting like, are we supporting just Prime Day in America, or are we supporting Prime Day all over the world? And that makes a difference in how you build the system.

Dialectic Thunderstorm: Okay, that, that makes sense. Okay, uh, so let me, let me start on non-functional with that in mind. Um, so you already said that, you know, we don't want to oversell. We, uh, that's one of the biggest concerns. So I— what, like, my immediate thought here is that we want consistency over availability. Uh, sort of like, we can take the hit to the, the UX if it means that we're not accidentally saying that we have something that we don't have and allowing them to go through that process. And then you said a million customers, um, and I'm sorry, was that a million customers a day that we said?

Declarative Dragon: No, it's 1 million orders per minute.

Dialectic Thunderstorm: Oh yeah, okay, that makes a lot more sense. 1 million orders per minute, so that is a very high rate of orders. Um, and then we— yeah, so are we talking about— are these 1 million orders, is that pretty geographically spread out, or are we just Kind of worried about one region?

Declarative Dragon: Let's say it's all over the world. Amazon has invented a new holiday called Prime Day and they're trying to make it happen. So they're doing sales all over the world at the same time.

Dialectic Thunderstorm: Okay. Okay. And are we— this million orders per minute, is this specifically during Prime Day or do we expect a spike above that on Prime Day?

Declarative Dragon: So this is just the Prime Day spike.

Dialectic Thunderstorm: Okay, okay. Would it be a good idea to ask for general, like, are we worried about on other days, what's kind of the average traffic we're expecting, or is this just a Prime Day thing?

Declarative Dragon: So the scenario I outlined started with on Prime Day, you know, like, so that's really the question is like how to handle a spike of orders on a specific 24-hour window, right? So if you can handle a spike, you can probably handle normal, you know what I mean?

Dialectic Thunderstorm: Okay, yeah, right, that makes sense. Okay, so, uh, this is— I don't know if this fits cleanly into non-functional requirements, but like knowing what I know about Prime Day, one thing that we see are lightning deals where we're going to have very high contention on single rows. So that's something we want to, um, manage high contention on single rows for Lightning deals. Is that worth putting in the non-functional requirements?

Declarative Dragon: I mean, it kind of just falls into the, I don't want people to be able to order things that don't exist, right? And then there are some things that are gonna be even faster than other things, right? So like a follow-up question to the question, if you can get through the workflow is, what do you do if two people select the same deal and put it in the cart in the exact same second? Like how do you handle that contention?

Dialectic Thunderstorm: Okay.

Declarative Dragon: Which is like, it doesn't, it, again, you like don't have to move the workflow out. The problem is self-contained in this like very small piece of the workflow.

Dialectic Thunderstorm: Okay, yeah, that makes sense. Okay, so I, you know, I'm wanting to move on and get to drawing boxes. Normally I think the next two steps that I have learned to do are outline the models and outline the APIs.

Declarative Dragon: Yeah, yeah, that's fine. Yeah, because usually it can go either way. People either do that first or they do the boxes and then immediately say like, okay, this box is a database, this is what I put in it, this box is the API, this is what I put in it. So it's, you know, six of one, half dozen the other. It doesn't matter which way you do it. So if you feel more comfortable doing the models and API first, that's totally fine.

Dialectic Thunderstorm: Okay, so what I'm going to do is just do like really really sparse. Like, I just want to outline what the models are, and then we'll get into the fields once I start drawing. Um, I mean, the most important thing I think is that we are going to have, uh, what we'll have products, uh, which, um, that is how we're going to manage, uh, you know, lookup of the individual product itself. And then I'm trying to think of what kind of table— like, we're going to have a general inventory table which is going to have information for each product. And what I'm thinking is we will have— just kind of taking notes here for myself— we'll have total amount and reserved amounts. And that's kind of something we're going to use to measure availability. That might change later on. And then we're going to have an order that's going to The reason I want to make this its own model is I think it's going to be stateful. Orders will be in different states, you know, they'll be in the cart or they will be the, the person has paid and the order is completed. There might be other ones that I, that I think of for that. And then I don't know, I mean, I know we want a user because we want to associate orders with users. I think that's mostly going to come into play when, when we do like We might rate limit based on user ID, things like that. I don't know that we really need to hydrate that model out that much. Do those make sense to you?

Declarative Dragon: Yep.

Dialectic Thunderstorm: Okay, great. And then I don't— like, I'm not really sure. I think it's worth— like, our API is really going to be like POST orders, they create an order. I'm just trying to do this super quick because I don't know if this is the real crux of the question. They can patch an order that they've already exist— that they've already made. They can delete. And we're going to have, like, get items, which will have our availability, which I'll get into more later. But— and then we're probably going to have POST orders, order ID checkout. Which will manage the last part of that lifecycle. So create an order on line 27, add items to the order on line 28, and check out with that order on line 29. Mm-hmm. Is that reasonable?

Declarative Dragon: Yeah, one thing I want to point out here is that you started to make a technical decision, and then you should be expecting that they'll always ask your reasoning behind any technical decision. They might not ask on this one, but it is one that I want to make sure that you handle and aren't like surprised by. So I might ask a question like, if it feels like we're gonna have time, I ask a question like, why did you choose REST here over other API designs?

Dialectic Thunderstorm: Right?

Declarative Dragon: Like what makes this a good choice? What makes REST a good, the right choice for this? Right? And then that's just an example of a question that I could be asking throughout the process. They almost always do it for database. It's like a number one. But any, anything, any basic, any technical choice that you make, when you're just like, okay, this is, you know, an SQS queue, or like, this is this. I'll ask why you chose that instead of something else.

Dialectic Thunderstorm: That is super helpful. Thank you. So the honest answer is that, you know, I just, I know REST, and I've heard like in system design interviews, REST is what you go with because it's tried and tested and usually the best option, better than something more complex. And like, I think this makes a lot of sense when we're talking about inventory. We have a lot of nouns and we're just acting on nouns. So I think the access patterns are pretty simple. We don't want to bring in a bunch of other complexity with something like GraphQL. You know, I don't really see a use case for that. We don't have rapidly evolving schemas or different uses depending on the query or anything like that. So I think REST— I don't really see a reason to go to anything besides REST.

Declarative Dragon: Okay. I mean, I also accept a— like, this is the one I'm most familiar with, so this is the most comfortable one I'm doing an interview. I think that is also a very reasonable answer, right? Just like, you know, your answer was fine. I would just say like, yeah, as long as you have an answer ready to go and you're not surprised that question came up.

Dialectic Thunderstorm: Okay. Okay, great. Yeah. I, yeah. And I appreciate that. That'll be harder to answer when I get to other technologies later on. So these are, this API, part of the reason I wanted to do this really quickly before I start drawing is I think this is how I'm going to want to model the, you know, the boxes and lines that I'm about to draw. I want map out creating an order that is probably like in the reserved state, adding things to that order, and then checking out and completing the order.

Declarative Dragon: Okay.

Dialectic Thunderstorm: Okay, so toggle whiteboard. We're gonna start with the client here. I just double-click client, and then this is, this is where I usually get myself into trouble. I I just had one of these interviews last week where I immediately stuck in an API gateway because I knew that we would be talking about rate limiting and things like that. But then it felt like there was some pushback on including that so early. So what I'm— the first thing I'm going to do is immediately just— we're going to hit an inventory service.

Declarative Dragon: What was their pushback on having it so early? Like, pretty much everyone does it, so I'm not surprised to see it early.

Dialectic Thunderstorm: Yeah, it was a little— I, I'm not really sure. I guess it was more like I made the assumption that he would understand that in something like this we wanted rate limiting, we wanted load balancing. But it kind of goes into what I brought up earlier where we're talking about a really simplistic design choice right now. So we don't technically need rate limiting or load balancing. I guess one thing I could put it in for this right now is we definitely want authorization for the user to add things to their cart. So maybe I could put in gateway and say that right now what we're handling in it is just auth.

Declarative Dragon: I mean, I would assume a load balancer and a security gateway for every service anyone has ever built ever at this point. So like, I, I would say I disagree with that interviewer, which is an interesting thing that comes up is like pretty much every interview I've seen, everyone throws in like an API gateway or a load balancer to start just to like show that they know that it's necessary. So I think it's interesting that the interviewer pushed back on that. That's— that actually just surprises me. They like say like, that's too soon, right? Like, I don't think it's too soon. I think you'll eventually get there. So what does it matter whether you do it now or later?

Dialectic Thunderstorm: Okay. Yeah, I appreciate that because I did feel like that was weird. So that's good to know. I will continue doing this in my system design interviews.

Declarative Dragon: Okay.

Dialectic Thunderstorm: Cool. Okay. So yeah, right now we're going to the inventory service and the exact request we're doing is the POST orders. So they're creating an order. And what this order does is it says to the inventory service, I want to reserve these amounts of these items. And here is where I'm trying to think, is this the point when they add it to the cart? Because they'll see like if one is left before they add it while they're browsing, but once they add it to the cart, I think this is where we need to do a check on do we have that. And I can't— I don't know how to do the, the very obvious DB shape on here that everybody uses for DBs, so I'm going to use a narrow rectangle. So this is our table, and I think what I called it in that other screen was just inventory, which is going to be products, multiple products. So the inventory service, it has to— it gets this POST orders where in the body we have we have product ID and amount requested.

Declarative Dragon: Mm-hmm.

Dialectic Thunderstorm: And— oh man, sorry. Okay. So yeah, so the inventory service is going to say, okay, we go to the inventory DB, we have this product ID and this amount requested, and we just make sure that we can get that amount. And if so, I think that in my opinion we want a separate DB for orders. They're a little bit stateful. We're going to be making updates to them. So the inventory service kind of works back and forth with the inventory DB. It says, yes, we can make an order. Oops. It adds that order to the orders DB where it's keyed off. We have the user ID, the product ID, and actually I'm not really sure how I want to model multiple products for the one order. I guess the order model is something that I need to think about.

Declarative Dragon: But once we— Yeah, I'm going to actually pause you here to This is something I see all the time, is just sticking with one service that does everything. I think you want to think about distributed systems, and that shows that you understand that at a big company, each service is going to probably do a very limited thing because you want to have a small blast radius to problems, especially at the scale of a million customers a minute. So having one service that it handles the inventory and handles the orders seems like a big ask, right? Like I would probably have an order service and an inventory service so you can really keep things separate. And I would have it where it kicks off an order first so that it has a staple order pre-check. Like as soon as the user is scrolling through products and maybe like create an order, and then once they, choose something and put it in their cart, that's when you can say, okay, let's add the products to the order and add the reserve to the inventory at the same time.

Dialectic Thunderstorm: Okay, so you think create the order once the user starts browsing and then add items one by one as they add them.

Declarative Dragon: Yeah, the way— the reason why I say that is because when you have to deal with so many things the customer is doing, you want to be as fast as possible. I'm gonna give you a great example on that for e-commerce, if you didn't know this. So when you start to type in your credit card number, they are checking it even when you've only put in a couple digits. Like they're double checking the first 4 digits before you even finish typing the rest of it, so that by the time that you put in the whole thing, they're done some of the checks and then they only have to do like one more check, right? And that's the idea of like, it's doing stuff as early as possible So that the user can have a speedy feel to their checkout process.

Dialectic Thunderstorm: Okay, yeah. Okay, that makes sense. That's very good to know. Okay, so I think then the first thing that they're hitting is the order service and we're going to create the order. Then that's going to create the order in the order DB. I think it's going to interact with it directly. And then yes, it is also going to go to the inventory service. This will be when they add items and it will say, hey, they've added this item, check that it's available. If it's available, you know, decrement the amount that we have available by what's reserved in this order. And the inventory service will say yay or nay.

Declarative Dragon: Okay.

Dialectic Thunderstorm: Okay. And then so at some point They have everything. I'm doing a simplistic modeling right here. So this is— they've created their order, and this also kind of handles the flow where they update their order with items at some point. They have this order that's in progress. The status is in progress, which I don't know if that's— if it's worth writing out this, but kind of just to think about the statefulness in progress or in cart or something. And then eventually they go to the checkout flow, uh, which that will definitely be a different service. And then I wonder if in addition to a different service for checkout, we want a different service for, um, payments to interact with a third-party payment service, assuming we're using a third-party payment service.

Declarative Dragon: Yes.

Dialectic Thunderstorm: Okay, so, um, I think what they would do is from the client they would say checkout, and then our order service, the checkout will be a POST. I think this is how I modeled it in the other thing, but we'll have go to this order ID checkout, POST with relevant information. When the order service gets that, it gets the order with that ID from the orders DB and it sends it to the checkout service to check out. And then I don't know if we want to have an additional inventory check here. I don't— maybe it would be a good idea to— I don't want to optimize too early, but future thoughts, maybe we would have a cache here where we could just do a really quick check. But cache wouldn't really— it would be hard to make that give the most up-to-date information about the order. Or about the inventory. So maybe we just hit the inventory DB again. I'm not sure about that yet, but we go to this checkout service, uh, the checkout service starts the checkout and, um, it goes to this third-party payment service. And I think when I thought about this in the context of [REDACTED], it's been a situation where we want to put a hold on the credit card, like what you see with [REDACTED], DoorDash, things like that. And then if there are any refunds or anything, we don't have to get a refund from the credit card company. But either way— oh, and I'm trying to think of if it's worth having a separate payment service. I think I'm trying to think of what the checkout service needs to do besides just routing from the order service to the payment service. So I don't see a reason to have a separate payment service yet, but I reserve the right to change that. And this might be a good candidate for a queue just because, like, what if something is wrong with the payment service or with the third-party payment service? We want to be able to handle that without breaking down something in our workflow. So then once they do it, I think that we will have a job that will go through the orders DB. And I'm getting a little bit into optimization here, so I don't want to do that too much yet. We'll have a job where maybe in addition to the orders DB, we feed things to an orders queue and we can say, okay, this order has been alive for 15 minutes. Go to the orders DB by that order ID that's on the message. See if it is marked as completed by the checkout service, which that would probably—

Declarative Dragon: Let me pause you here. I think this happens as well pretty often. So it doesn't surprise me that you're doing this, but you're doing a lot of talking and not as much drawing to match your talking, and it's harder for me to follow things along. The number one thing is like a lot of this like posting and stuff, you're not writing about like what data you're moving along, and even if it's very simple, just like, oh, we're posting an order ID with the number of, you know, and then the order service sends the number of products and the total cost to the checkout service, right? And then you're talking about other stuff happening and it's just not, it's not being preserved in the, in the design. So it's harder for me to like follow along, you know, just as someone that's like, you know. So it's just always good to like, even if you're just drawing very simple squares, just be like, all right, I think, or just like writing in like, here's, you know, all that is passing along is the payment total. Because what it looks like is the order service is kind of like the control plane of calling out the other services. And then if some— one of the other services fails, it's going to kind of completely unravel. So like, that's why I kind of want to understand everything that's happening here that you're talking about that I'm not seeing, right? So like, if you want to make a decision now, just like make a decision and put it in there, and then we could always— you'd always change it.

Dialectic Thunderstorm: Okay, yeah, will do. So let me start fleshing out what I'm thinking a little bit more because I think that's something really good to practice. So on POST orders, add order to DB. And it's going to— what's it going to look like? That just creates the order. So we'll associate it with a user ID. I mean, I want a last updated at. So I'll just put in a created at. But I think updatedAt would be good to have some kind of a TTL on orders. So that is what the order service sends to the orders DB when they post the order. And then when they update the order, let me think.

Declarative Dragon: Oops.

Dialectic Thunderstorm: Okay. I've broken things. Okay. So the order service, it creates the order. When the user put— or, uh, service will patch the order at that order ID, and it will be, you know, all of those fields already— well, we'll update updatedAt, but we will add A product, which I'm trying to think about how best to model a product within an order and if it needs to be its own model like product order and then we can—

Declarative Dragon: What about just product ID and number?

Dialectic Thunderstorm: Okay.

Declarative Dragon: I mean, that seems a simple—

Dialectic Thunderstorm: Yeah. Okay. So yeah, sending that to— on a patch, we do that. And then what does the order service do when we go to checkout? I won't do that yet. First, I'm going to do the inventory part. On this patch, the order service is going to go to the inventory service and check availability. And it will do that via the product ID and the quantity. It'll say we need this amount for this product ID., and the inventory service will go to— this makes more sense over here. The inventory service will go to the inventory DB and say something like get inventory quantity or something, but for product ID and quantity, and it will return that we don't have enough or we do. Oh, that probably shouldn't put— that shouldn't be that. That should be like a— that should put a hold on it. And maybe at the application level we will return an error if it can't put that hold on it if the quantity isn't sufficient. Um, okay. So then if the inventory service gets back to the order service and says Yes, we can do that. Then the order service continues on, and then when we go to checkout, we will say—

Declarative Dragon: I'm gonna pause a second. What do you mean that order service continues on? Does it make a change to the database saying that like Like, did it add the product ID and quantity to the database before it checked the inventory, or is it doing it after it checks the inventory?

Dialectic Thunderstorm: Uh, I was thinking having it do before it checked inventory so that I, I guess that would mean another DB write, but then we could— like, I, I'm thinking about how to give the user the immediate feedback item added to cart. Um, but yeah, maybe, maybe it would be better to hit the inventory first and come back to them and say I mean, it's fine either way, as long as there's something in the order that is clear.

Declarative Dragon: Like you have the calls here, but you don't have what you're saving in the database maybe. So maybe if you have the state, either you could have more clear states like inventory check, or you could have some another field. It really depends where you want to put it. But I would have something to make it clear that you did this check. And if anything checks the order database about an order, it can know that the inventory check was either not done yet or done.

Dialectic Thunderstorm: Okay, okay. So let me add, let me add a little bit of a model for the inventory DB or the orders DB. So a row in the orders DB will look like, you know, user ID, um, product Yeah, and this is where I'm having trouble thinking about how to model it, but what I'm thinking here is we have a product ID, a quantity, and like a can fulfill. But that, you know, that's where I was saying that maybe that feels like it needs to have its own model and then an updated_at. And I'm putting a created_at because it feels weird to have an updated_at without a created_at. And I, but I, the usage of the updated_at is we want a TTL on orders, especially with high traffic. So every time we patch an order to the orders DB, we add the product and a quantity. And then the can_fulfill is the thing that, that's the question. It's going to come from the inventory service. Is it too heavy to do that check to, like, write the order to the orders DB, assume something for can fulfill, and then make the call to the inventory service and get back to it and set whether it can fulfill it or not? But at this point, I mean, we have a record of the user trying to get that order. That might get us into trouble later. And then I want to map out what the inventory DB is going to look like.

Declarative Dragon: Well, what about your status? Is that still in the database?

Dialectic Thunderstorm: Oh yeah, you're right. I apologize. Status, and I'm thinking that will be something like in cart, in checkout, paid, or completed or something.

Declarative Dragon: Yeah.

Dialectic Thunderstorm: And then the inventory, I think the main thing with the inventory is it's going to have like a collection of product inventory which is going to have product ID, total reserved, and then available which will just be a function of total and reserved. So yeah, when the— that'll be the easy check that the inventory service does. It will say, hey, for this product ID, we're indexed on product ID, I have this quantity that I want to send. Do we have that available with total minus reserved? And then that will get back to the order service. It will say success, and the order service will say can fulfill true. If it's not success, can fulfill is false, and then I have to figure out how we want to handle that. It might be like a checkout time check, but this would be sort of an optimistic blocking where we show the— or optimistic contention or something where we show the user that they can fulfill it, it's been added to their cart, and then we we tell them at checkout time that that can't happen. So that actually— so I think since we're creating the order first with that POST, what I do want to do is, yes, I want to hit the inventory service before the order is updated and on success update the order with that product ID. And then that would mean we don't have a can't fulfill because it's not added to here unless we can't fulfill it. And then we can immediately get back to the client and say if fail, you know, alert in UI.

Declarative Dragon: Okay, let me do one fix to this. Okay, so say I asked for 5 apples and you have 3 apples. What do you want to do? Do you want to say fail? Just fail? Because that doesn't give me all the information available.

Dialectic Thunderstorm: Yeah, that's true. Okay, so, okay, so I think what we would want to do is they would reserve the amount that are available. So the response from this patch, in addition to just success or fail, it would be amount available. It would have amount available. So let me write that out. Response. Can fulfill, which will be a Boolean. And if it's false, we'll have the amount available, which will be, you know, some amount smaller than, smaller than what was requested. And that will go back to the inventory service, which will It will send this response to the order service and the order service will update the order. It will have maybe instead of like a fulfilled boolean or something, we have the product ID, the quantity, and like quantity requested and quantity fulfilled. Then that gives us a really easy, we have this DB value that we can use to get back to the client and say, for this, uh, for this order update you just made, uh, for this product ID, you requested this amount, we have this amount, and then we get back to them immediately.

Declarative Dragon: Okay, um, since we only have 13 minutes yet, I'm going to start what I always do on these questions when we pretty much got the workflow working is I start asking about failure modes. So like, what if the inventory service fails? Okay. Or like, how do we want to handle making sure this works in Japan and Brazil at the same time? So those are like my main type of questions. But then you talked about having like event queues and things like that, because I think here you— this is pretty much a complete synchronous service and that it might involve some slowness to the users. So I think that you were going to the right place, which is like, how do you make sure this is— another question to include all this is like, how do you make sure this is fast as possible? It's as resilient as possible and it works everywhere in the world, or kind of like the three main categories of questions. But I usually do very specific, like, okay, what happens when the inventory service crashes? Or like, what happens when the database goes down? Like, I will pull apart all the pieces and say, What do you do now?

Dialectic Thunderstorm: Right. Yeah, okay. I'll start with what happens if the inventory service crashes. The effect on the order service is it doesn't get that inventory response back. The question is, do we want to be— the original requirement was we don't want them to be able to buy things that don't exist or double charge them. But should they be able to add the thing to the cart? I think that is reasonable. I think we should have availability for adding items to your cart. Even though it's a degraded user experience if they get to checkout and they see, hey, you had this in your cart, now it's unavailable. This would be assuming at some point the inventory service comes back up and they go to checkout. I think that that is acceptable. It's better than just not being able to add anything to your cart because the order service can't contact the inventory service. And I wonder if— so the question is, how do we want to handle not getting responses back? And do we want to do that via a queue sharded by order ID? Which, yes, I'm not writing it because I don't know if it's the right answer, but I will go ahead and diagram it out. Order queue where message— well, the message ID is the order ID. I'm thinking of like Kafka where we use Kafka, this only seems to be going to one place, but partition on the— partition on the order ID. Yeah, okay, so what the order does— oh, these aren't together— what the order does is it will kick off— see, I'm kind of going back and forth here. I wonder if we want to order update message from the order service, which goes to the order queue, and then we have like an inventory worker.

Declarative Dragon: Oops. So I'm going to pause here and say I love event queues. I feel like they're one of the best ways to handle the scope. I would also say that you can say something like, well, I understand the inventory service could fully come down, but one of the ways we keep it resilient is making sure that it's horizontally scaled so much that, you know, and that you're using the sharding on the message ID so that if one of them goes down, it automatically another one pops up in its place. And while it's coming up, the messages go to a different Inventory service, and that's a way of handling— Inventory service is less yours, but if I said like the order service, like the instance goes down, you could say like, well, it's okay, because I have it set up to be horizontally scaled with all the— everything saved in messages so that it can immediately go to the right service, the right instance, a different instance as soon as it comes up, right? So that's really helpful with the regionality. I always talk about just basically copying the service to different regions and splitting the calls. You know, you pick like a country ID or region ID, you know, you kind of base, since it's Amazon, I would say to base it on what their availability is, all right? So like, you know, you could base, the, you know, US-East-1 gets the US-East, you know, is a service in every single availability zone. And then the orders in that availability zone goes to that copy of the service. And if one of the services completely goes down, you could have a backup zone that's a little further away, right? Things like that to just make everything more resilient when you're kind of at the Amazon size. And I wanted to get, you know, give you that information before you get too far down the road of like, You know, you're going in there. I'm telling you, you're going in the right direction. It's just like, make sure to add all that information about like how you would keep this from just being, you know, a simple e-commerce site that only works for 100 people.

Dialectic Thunderstorm: Right. Yeah. Horizontally. Can you hear me?

Declarative Dragon: Yes.

Dialectic Thunderstorm: Okay. Sorry. I was saying, yeah, I worry that it sounds too easy to say, oh yeah, well, hey, if we want, if the inventory service goes down, it's fine because we've sharded horizontally by region. We have different inventory services based on availability zones. If one goes down, they just hit one further away. The question for me then is, how do we want to handle the DB if inventory service in Washington goes down and it's hitting the Washington inventory DB for, you know, that there— I don't know, there's some regional indicator of which inventory we're looking at. Is it acceptable that if they're then going to the California inventory service, do we just hit— I guess we would just hit the California inventory VBs, but then we would have to keep track of the fact that fulfillment comes from a different place. So I think knowing our stock breaks down a little bit there, and I don't know how to handle that.

Declarative Dragon: Yeah, um, I think you want to think about— you know, you said it sounds like an easy answer. The, the easiness of the answer depends whether you own the service or not, so you can ask that question. Like, can I completely control the inventory service or is that owned by another team and I get, you know, a nail, or there's another team that owns like the main inventory service and this is like a cache that I control. 'Cause if you control, you can use message queues and you can, you know, have horizontal scaling and all that stuff. If you have no control over it and it's just something you're hitting, say like the third-party payment service, that goes down, you have no control over it. Say you're using Stripe and that goes down, No control over that. You know, you just have to think about what is the best user experience for that and be like, hey, we're going to hold your inventory for you and make sure that, you know, things are, things are coming back as soon as you can. And then you can also do things talking about bringing things forward so you can kick off as many events as you want, right? So you could kick off events and say, check their current credit card, see if they have a credit card on file and the main credit card on file. Does it have— can I, you know, put a hold on it for what they currently have in their cart before they even check out just to see what's available.

Dialectic Thunderstorm: Right.

Declarative Dragon: Right. Things like that. You can just like check ahead. You can check inventory ahead. You can check checkout ahead and just like, you know, when they're looking at an item, you think about like, okay, let's check one. Okay. One is available. Okay. Two, you know, like just kicking off all these events. As long as you have enough, if you're willing to pay for the services, and the compute, you can make the user really well. So, okay, so does that make sense? By the way, I'm gonna pause there for a second.

Dialectic Thunderstorm: Yeah, no, I think so. When you say kick off the events, are we talking about asynchronous events that we just do in the background and they help inform what we do when they actually go to make a decision?

Declarative Dragon: Yes.

Dialectic Thunderstorm: Okay.

Declarative Dragon: Yes.

Dialectic Thunderstorm: Okay.

Declarative Dragon: Yeah, then you, then you'll have more information when they go to like make a decision about what they want to do.

Dialectic Thunderstorm: Yeah, that makes sense.

Declarative Dragon: Okay, and then you're not— you're also— the nice thing about events is you don't lose information. So if you're just hitting an API when the service goes down, that API request just kind of gets lost in space and time, right? And let's save it as a message to be processed, then messages can always have dead letter queues and things like that. And so at the speed and size of this service, that's why I like, okay, I'm going to create this API, and I would say this design works really well for a small service, but I'm going to break it apart and put message queues in between them so that I never lose a request If something, you know, if an instance, you know, if I have 1,000 instances, one's always going to be going down. So like, I don't want to lose requests, right? I want to save them somewhere so that they always, you know, so even if the instance goes down, it gets to the right place.

Dialectic Thunderstorm: Yeah. Okay. So it does seem like when in doubt, the answer is usually message queues because they're resilient and they'll get it there eventually.

Declarative Dragon: Yeah. And I mean, I usually do what you did, which is like start with the APIs and you can show the workflow kind of happening. And then you think like, okay, how do I make this more resilient? It's kind of the next question. That's the next level of it. That's why I want you to like draw this out and have things written down as soon as possible. And you can see like there's still like so much more we could do. Like we have, we don't, we haven't talked about like incremental backoffs for some of these questions or like other, other things that we could do. And we're already like out of time. And that's why I want people to start drawing as fast as possible because sometimes people send text 20 minutes in the original question and I'd be like, please stop, please go to the boxes. It's more like this, this tiny thing is more complicated than you think. It's always more complicated than you think. So like, let's get to the tiny thing as fast as possible.

Dialectic Thunderstorm: Yeah, I think that is phenomenal advice and I appreciate that.

Declarative Dragon: No problem. All right, 2 minutes left. Any questions about any of the feedback I've given you or just general any questions about system design? That I can answer for you and like how I do it. I've done like so many of these. I had to go and count for Interview.io and I got like bored of counting after like 40.

Dialectic Thunderstorm: So like, I, I like the questions about the actual process and things I can ask offline, you know, look up and things. I really just, if you can give me feedback on generally my performance and what I could look for besides getting to the boxes faster that I could improve. Yeah. I, I thought get to the boxes faster and write more down.

Declarative Dragon: Yes. The other thing is make decisions. 'Cause I think you were like waffling a lot. You were like, okay, we could do this or this. And then you weren't sure, right? Like one of the things about system design is it's usually with the level of a senior engineer. And I always tell people like, if you're gonna be a senior engineer, you have to be able to make a decision and then go back and change it if that decision was wrong. But at least start with a decision saying like, okay, we could do this or this. Makes it seem like you're, you know, like just pick one and then if you're like, start to do it and you realize, oh, you know what, this is wrong, just go and erase it, you know, don't, 'cause like when you're saying this or this, I'm not sure what you're just actually deciding on, you know.

Dialectic Thunderstorm: Yeah, yeah, that's great. I know a lot of the feedback that you read is like, understand that every decision you make has trade-offs and I try to verbalize what those are, but yeah, that prevents me from actually making the decision.

Declarative Dragon: Yeah, and like, I don't think you talked a lot about the actual trade-off though, like if that's what you were going with, 'cause you kept saying like, we could do this or this, but you weren't saying like why. You're like, okay, we could do an API call, which would synchronously handle this, and you know, which is great, 'cause then the information is, it's a simpler system, or we could add an event queue, which would handle it asynchronously, which means it might be a slower response time, but we are less likely to lose anything if an instance goes down. That, you know, you can see that trade-off and then you can say like, okay, so I'm going to go with an event queue because I'd rather the user get, you know, the right— always get the right information eventually. Because as you talked about in the beginning, right, like availability versus consistency.

Dialectic Thunderstorm: Yeah, yeah. Okay. Okay, that's extremely helpful. Thank you.

Declarative Dragon: You're welcome. All right, we are all done. Good luck at [REDACTED]. I would definitely recommend doing at least one more where you where you say, hey, give me the feedback at the end, and have them do a regular style interview and see how it goes for you. And you might need to do a couple of those. But I mean, I think you're getting there. It's all about, you know, there's two things, right? There's having the information and being able to express it in an interview are two very different skills. And it sounds like you have a lot of the technical information. You just need to make sure that you can structure it well for this interview, which is very different than real life, right? In real life, one person is not building anything this big. You know, like 20 people are building a fraction of the inventory service, you know, in real life. So it's kind of a puzzle, but if you're working for a startup, you might build the whole thing. So it really depends on what you're doing.

Dialectic Thunderstorm: Yeah, yeah, okay.

Declarative Dragon: Cool, all right, good luck.

Dialectic Thunderstorm: Thank you very much, have a good rest of your day.

Declarative Dragon: You too, bye-bye.

Dialectic Thunderstorm: Bye-bye.