We helped write the sequel to "Cracking the Coding Interview". Read 9 chapters for free

Design an A/B Experimentation Platform

Watch someone solve the design an a/b experimentation platform problem in an interview with a FAANG engineer and see the feedback their interviewer left them. Explore this problem and others in our library of interview replays.

Interview Summary

Problem type

Interview question

Design an experimentation platform that lets internal teams split user traffic between control and treatment experiences, keep assignments consistent over time, collect outcome metrics, and support both classical experimentation and ML-driven optimization.

Interview Feedback

Feedback about Dapper Badger (the interviewee)

Advance this person to the next round?

How were their technical skills?

2/4

How was their problem solving ability?

3/4

What about their communication ability?

3/4

TLDR; You handled ambiguity well and asked great clarifying questions early. I do expect a senior to cover more ground on things like scale and extensibility but based on your area of expertise I'm not surprised you spoke most in depth around the analytics component (which is great). To take your system design to the next level, anchor questions in the user experience (“how would a customer interact with this?”) to ground the design. When asking questions, briefly explain why they matter or what tradeoffs you’re considering as it signals depth of thinking. Early in the session, outline your agenda (functional + non-functional areas, time allocation) to guide the conversation and steer the convo. As you design, call out key considerations like monitoring, extensibility, and data storage tradeoffs, even if you can’t go deep, mentioning them shows awareness and then gives you a nice list to refer back to if you have bandwidth / time. RESOURCES Some system design notes: - ask yourself, how do I know that my system is working properly - talk out loud - talk about the pros and cons of an approach and why something isn't optimal - evaluate - observability - extensibility - maintainability - availability - scalability - security / resiliency / threat modeling For more practice I really love Gaurav Sen's videos on YouTube: https://www.youtube.com/watch?v=QzLhb1WBFjQ General good content: https://www.interviewbit.com/system-design-interview-questions/#sql-vs-nosql Design YouTube: https://bytebytego.com/courses/system-design-interview/design-youtube?fpr=javarevisited Design a chat system: https://bytebytego.com/courses/system-design-interview/design-a-chat-system Design a URL Shortener: https://www.educative.io/courses/grokking-modern-system-design-interview-for-engineers-managers/design-and-deployment-of-tinyurl Things I share for behavioral sessions: To re-iterate some high level data points to make sure to hit when you are sharing examples: * use data and quantify wherever possible -- timelines, deadlines, scope, visibility, customer, customer size, impact, etc. this helps give your interviewer a better understand of the scope, complexity, and situation * set the stage with your role and responsibilities -- i.e. if you were tech lead, if you were also leading from a PM perspective, if you were working directly with customers and gathering feedback * how you were helping to prepare / be operationally ready -- putting metrics in place and utilizing two way door decisions * if you are stuck between sharing two different examples, ask a follow up question or ask the interviewer which example they would like you to dive into as they might have something specific in mind they are trying to capture. RAW NOTES / TIMESTAMPS A/B Experiment Platform :11 Prompt + functional requirements >> good clarification, I would use the framing of asking for a user journey / experience to ground the conversation Partition by user or session >> great question >> hint: talk about the why behind your questions Good question on metrics component >> helpful to give a high level agenda / walk thru of how you are structuring the time >> instead of asking if you captured everything, reframe :19 Non-functional good clarification on how to experience it :26 Starting to draw app/user Hashing for partitioning -great! Talk more about why this is a great solution Storage? Cache? :31 Complete pictures setting up databases - metrics database sql Separate process watching the user interactions here :41 Taking about data comparison (good clarifications here) Leaning in on the BI side (I expected this given your area of expertise) >> good strategy on talking about the missing components (this is where you can pull from my list) :46 Prompted on the bucket user component We need to store information on the mappings - more of a key value store :49 Feedback Things to touch on Security Extensibility (scheduling, multi-treatment) API calls Splitting out read vs write heavy dbs Scale

Feedback about Covert Gecko (the interviewer)

Would you want to work with this person?

Yes

How excited would you be to work with them?

4/4

How good were the questions?

3/4

How helpful was your interviewer in guiding you to the solution(s)?

4/4

Interview Transcript

Dapper Badger: Okay. Well, yeah, thank you for, for making time. I know it was on fairly short notice, so yeah, I appreciate that.

Covert Gecko: No problem. I, I always love doing these more senior system designs, so looking forward to jumping in.

Dapper Badger: Yeah, me too. All right.

Covert Gecko: Yeah. Talk to me a little bit about— on my end, I'm not sure if you've used the platform before, but all I can see is your pseudonym and that you want to do system design practice, and that's it. So I'd love to hear a little bit more about what you're looking for and what you're hoping to get out of our session today. And that way I can make sure to tailor it for you.

Dapper Badger: Yeah, sure. So I haven't done a systems design interview. I am more of like a, I started off really as a data scientist and then became more of an engineer kind of over time. And so yeah, I'm applying for a lot of like machine learning engineer roles. So I guess, yeah, one thing is like just having my very first experience with systems design interview. I actually have one tomorrow. Okay. Um, with one of those big AI companies, uh, for like a member of the technical staff role. Um, and so I think, you know, in some ways really just going through this process once would be like super helpful. Okay. And then getting high-level feedback, cuz there may be some, like, there So there's probably going to be some broad strokes things that I will need work on. I guess, you know, it might be nice to have something that maybe has a machine learning flavor, but I, you know, I'm not sure if the interview tomorrow is going to be like a machine learning systems design or more of just like regular software engineering systems design. So I guess I actually have another one booked after this, which is specific, like in 2 hours, which is specifically machine learning systems design. So I don't know, maybe it would be good to get a regular design. Let's just do—

Covert Gecko: yeah, you said this is for a technical staff role, is that right? Well, do you know what that means in terms of leveling? Like, is it a staff-level role or is it more just like a senior, um, like a senior role?

Dapper Badger: Not totally clear. I think it could be pretty wide range based on the salary range they gave. It was like, like 3, like all the way from like the, the minimum, the maximum was like 3 times the minimum. So I think there's a really wide range.

Covert Gecko: Okay.

Dapper Badger: Um, so it might include like all the way from non-senior all the way up to staff. Um, but I, I think probably they're looking at me as a senior or staff, not totally clear, but I think probably at least senior.

Covert Gecko: Okay, cool. Great. Um, the reason I ask is I, um, I usually ask this question of like my more senior and staff, um, group. So I'm just going to pretend and try to evaluate you at the staff level. And then I can give you specific feedback towards that. And that way we can chat at the end of like where I think you would net out, where, you know, where to kind of lean in on different things.

Dapper Badger: Sure.

Covert Gecko: And then, and then the question that I ask could have an ML component to it. So I think I'm curious to see where you take it versus where, where other engineers take it. So sure, that will be That would be great. I'll also drop, I drop these in the notes at the end, but I'll drop them for you now just to kind of have some context. But these are just some like general things I've personally found. Like I found all of this context for myself going through my last system design interview a few years ago. So I always try these out as some helpful things that I personally found while I was in your shoes.

Dapper Badger: Okay, great. Do you mind if I like copy and paste this somewhere?

Covert Gecko: Oh, please do. Yep.

Dapper Badger: Okay.

Covert Gecko: I will paste this in your notes too, so don't feel like this is the end-all be-all. I will drop it in for you too.

Dapper Badger: Okay, awesome. Thank you.

Covert Gecko: Yep.

Dapper Badger: Yeah, I've seen some of these videos, I believe, but I don't think I've seen like Gaurav Sen's videos. So yeah, that should be really helpful.

Covert Gecko: They're pretty incredible. I found him just kind of searching for some support in terms of prepping for system design. And the thing I really like about his videos is he has videos. It's like, okay, let's go through the problem, let's design Twitter. And he will talk to you about like how to answer the question strategically. But then also if you get to a point where it's like, okay, do I use a SQL or a NoSQL database? He actually will then have these like spinoff videos for those kind of like linchpin decisions. So I found it really helpful because then I could go watch the other videos and kind of like fill in.

Dapper Badger: Okay, oh, that's cool.

Covert Gecko: How should I have made that decision? Um, that I personally found really helpful for myself. Um, so I always share it with people on here.

Dapper Badger: Yeah, that's, that sounds great. It's kind of like branching. That sounds a good way to study for sure.

Covert Gecko: Exactly. Um, have you done another interview on this platform before?

Dapper Badger: Um, no, I haven't.

Covert Gecko: Okay, so just kind of give you, um, kind of kind of high-level, we'll have our session today. At the end, we fill out feedback for each other. So this is where I will drop these notes for you. I also personally take timestamped notes once we start the system design. That way you can kind of track where are you spending too much or too little time as well. And you can actually replay our whole session. So you can go back and say, okay, I need to look back at the, the 19 and see, like what kind of feedback was happening at that point. So don't feel like stressed to try to take notes when I rapid-fire feedback at the end. 'Cause I try to give you as much as I can in a short period of time. So I just don't want you to feel like you have to write everything down 'cause you can go re-listen to it.

Dapper Badger: Okay, that is good to know for sure.

Covert Gecko: Yeah, yeah. Yeah, and then if you have any other questions, I always unmask at the end so you'll get my contact info if that's something you're comfortable with. So don't be shy, those are things I can help with.

Dapper Badger: Okay, that sounds great.

Covert Gecko: Cool.

Dapper Badger: Definitely.

Covert Gecko: Great. And then how much time do you have for your system design tomorrow?

Dapper Badger: Yeah, it's a good question. So the person I'm talking to is kind of the head of the group. He was like the person who initially screened me as well.

Covert Gecko: Okay.

Dapper Badger: And so he's pretty senior and I think he may want to ask me some other questions. Like I guess I would just be surprised if it was purely technical, which means Probably the systems design will be like 45 minutes or even like, um, if I wanna include some time for like deep dive or improvements or stuff, I, I might wanna shoot for like half an hour to 40 minutes.

Covert Gecko: Okay.

Dapper Badger: Maybe. Great. Yeah.

Covert Gecko: Why don't I give us 35 minutes? I'll set my clock. Okay. Um, again, I'll take some timestamps for you. I, I'd like to kind of con— like, you know, kind of go with the lower end of your budget that way. Were kind of working on your time management skills. And then also if you feel like you needed more, like, oh my gosh, I needed 10 more minutes, what would I have done? We can talk about that too. If you feel prepped to handle a short period of time or a longer period. Yeah.

Dapper Badger: Okay. Okay. That sounds good. Great.

Covert Gecko: So I'm going to slow down at the 45 and then we'll chat through feedback. Um, there's a toggle whiteboard button on the bottom here. You can tap that and it will pop us into like kind of like the next FileDraw.

Dapper Badger: Cool.

Covert Gecko: I can see it.

Dapper Badger: Okay.

Covert Gecko: Yeah, cool. I've been noticing lately sometimes that I can tell the person is like typing or drawing or something and it won't show up on my end. So if so, I'll just interject and let you know and have you refresh your page, but hopefully we don't have any issues.

Dapper Badger: Okay. Yeah. Maybe one of these questions could be how to build interviewing.io or something. I know.

Covert Gecko: I know we should, uh, That'd be kind of a fun one.

Dapper Badger: Maybe they should craft that. Right. Awesome. Yeah, cool.

Covert Gecko: So let's jump into it. I am gonna purposely pretend like we haven't had this conversation. So I try to be a little bit more hands-off in these just to force you to kind of steer a little bit more. So the floor is yours. Today I want to design an A/B experimentation platform. I know, it's tough.

Dapper Badger: An A/B experimentation platform. Correct. Okay, interesting. So we, do we have any sort of specific types of applications that we want to be A/B testing? Like is it, yeah, consumer-facing or, Yeah, just kind of curious to know more about the use case there of like what types of applications we'd want to be A/B testing and like how flexible I guess we want it to be.

Covert Gecko: Yeah, good question. So let me give you a little bit more context and like maybe like a user experience example. So let's pretend that you're building this platform internally at Amazon and this could be used by the website and the frontend teams to be able to, you know, maybe show, you know, a Christmas content to 50% of users. Or we could be using it on the backend to, you know, slowly roll out a feature or shadow launch something.

Dapper Badger: Mm-hmm. Okay. Roll out a new feature. Okay. And so, okay, and do we, we have a use case for being able to have two different types of, two different types of websites? Maybe there's like different graphics, maybe there's, I mean, entirely new, entirely new sort of backend features. Okay, and Yes, we want to be able to— okay, gotcha. And obviously we would want to be able to, you know, direct some users to A and, you know, others to B. And this is— we want to partition this by the user, or do we want to partition it by the, like, session? Like, with the same user, if they came back let's say at a different time, would they still get option A if they got option A before? Or are we kind of partitioning this on users or sessions?

Covert Gecko: Mm-hmm. Yeah, um, that's a great question. Let's, um, let's try to make this idempotent. So we'd want the same user to have the same experience every time they come back.

Dapper Badger: Okay, gotcha, gotcha. Okay, and then I'm assuming, you know, we will want to be collecting some important data that would be relevant to the operation of the business. Are there kind of— how flexible is that? Do we have maybe sort of a preset kind of features that we would want to look at that we offer as a service, or do we allow them to maybe be able to define things themselves, or is it kind of selecting from a menu of things things like, you know, click-through rate and amount of time spent on the page? Or are there any, yeah, are there any parameters around the metrics that we want to collect on these two different options?

Covert Gecko: Yeah, that's a great question. I'd say we want to have kind of a set of things. I think the things you listed are kind of the usual culprits in terms of just like click-through rate, you know, checkout rate. But I'd also love to know what you have in mind in terms of like say I have a unique use case, what would your platform be able to support or not, or how could it be extended?

Dapper Badger: Okay, so there's an interesting question. Maybe we can expand on this, like iterating on it a little bit later, but how would we provide custom metrics? Yeah. Okay. And let's see. So those are some of the functional requirements that we have here. I'm trying to think of other, I mean, so we want a user to be able to log in and see the same thing depending on what their user ID is. And then obviously we want these metrics to be fed back to the person hosting the website or who owns the website. Do we want to go beyond just metrics? Like, do we, is, or is it part of the minimum product? Because, you know, there's kind of A/B testing and then there's like some kind of multi-armed bandit thing where it, you know, it might actually like adapt to which one is doing better and start, and start serving that content more often. Or is this more like purely experimental and analytical, but not necessarily like adapting and trying to push the more popular one into production, like straight away.

Covert Gecko: Yeah. You, you picked up on why I said this can be an ML question, right? Um, I would like to offer the functionality for either. So talk, you'd have to talk to me about like when we get to that point, talk to me about like, you know, where, where the decisions might change based on. Which path?

Dapper Badger: Does that make sense?

Covert Gecko: Right. Yeah.

Dapper Badger: Yeah, that does make sense. Okay. Just trying to think about, think about that. Yeah. Because, and you said 50% is like a common percentage. So this isn't, this isn't like a tiny sliver of users for like the experimental thing. It's kind of fully production, fully scaled. Both options are at full scale basically.

Covert Gecko: Oh, good question. Yeah, so I would want the ability for— say I want to ramp something up, maybe just expose it to 1% first, then be able to increment to say, all right, for a week or for a couple days, let's have it at 1%. Maybe I'll pop it up to 20%, um, after some time.

Dapper Badger: Okay, scale it up to 50%. Over time. Okay. Right. And that would obviously dovetail with if we wanted to automate that using machine learning, and that would be an interesting goal. Okay. And then in terms of, I mean, have I captured like the majority of the functional requirements here, do you think? We've got users who are going to see either page. We're collecting metrics. We can set the split. I mean, I'm sure there's other more advanced things like, you know, more than 2 options or something, but I feel like it might be safe to scope it down to just A/B testing for now instead of like A/B/C/D.

Covert Gecko: Yeah, exactly. Yep. Yeah, I think this looks good. Okay.

Dapper Badger: Okay. And then in terms of nonfunctional requirements, Do we have any— I mean, in terms of like the availability, you know, what kind of uptime are we expecting here for both of the pages? I guess— I mean, I'm assuming that it's similar availability requirements for both branches. Mm-hmm.

Covert Gecko: Yeah, same. Okay.

Dapper Badger: Okay. So basically all nonfunctional requirements are the same for both branches. OK. And then in terms of what they actually are, I mean, hmm, this is an interesting one because it's supposed to be able to work for kind of any web page. And I mean, it's— I'm just trying to think of— I mean, they're sort of serving two different frontends. But the frontend isn't gonna impact too much, like, the availability. And a lot of these nonfunctional requirements really would depend on the actual backend that that website is using, which we can't really control. Like, there's some things that are kind of out of our control here, right? Mm-hmm. And so it— yeah, I mean, in terms of how do we define those nonfunctional requirements, is it— we basically Do we want to define it kind of relative to what the— you know, it shouldn't make the uptime worse than what the original website was?

Covert Gecko: Yeah, like it shouldn't degrade the experience at all.

Dapper Badger: Shouldn't degrade the experience, right.

Covert Gecko: OK. Yeah.

Dapper Badger: Right. And all— right. So in terms of reliability and scalability and all of those things should be not not degraded. Scalability. Yeah. Redundancy. So, I'm just trying to think because some of these things are out of our control, like what database they're using. I'm thinking, okay, maybe there's a website that has, you know, a certain data model, a certain backend and all that stuff. And we're— we want to provide a different website. But we're not planning to provide them with like a whole different backend, really? Like we— because I mean, I suppose you— if you really wanted to create two totally different systems, that could involve having, you know, totally different data model and having to like run a data migration or something like that, or doing a lot of changes on the backend. Are we— is it okay if we kind of scope this to Basically, we're planning on just making that the— basically, the API on the backend is the same for both branches, but we're just interacting with it in different ways and displaying it in different ways. Yeah. And that's kind of—

Covert Gecko: yeah. My answer to that is yes, but I'm curious, talk to me about, like, the user experience here from, like, a developer perspective. Talk to me about how you're envisioning developers would use your system.

Dapper Badger: Yeah, OK. So I suppose if we have— I mean, right, there's got to be some kind of maybe like an API wrapper is kind of what I'm thinking. The system already has some kind of an API. And maybe we have some— basically, the developer or engineering manager has some new API version, let's say, or, you know, some new API calls that they're trying to test out. Then Essentially, we could have a service that sort of routes traffic to the API and kind of wraps around it. But it will make maybe, it'll make a decision about which path to take and which API call to make depending on which branch the user is on. But at the end of the day, we're still relying on a backend API that the developer has already made. It's basically, we're just a wrapper around that API. Does that make sense?

Covert Gecko: Yeah. So the, the, the, the basically the wrapper is like, you know, the developer would call and say, okay, is, you know, customer 123, like what treatment are they in for this experiment? And it would just return whatever that treatment is.

Dapper Badger: Right. Right, okay. And it would return and basically that would also sort of have a different interface. So, from that point on, if user A, I don't know, I'm trying to make something concrete. Like, let's see. Maybe there's one version where I don't know, we want when a user signs in and makes a comment that the comment, like, maybe it shows some new feature here that's like, I don't know, top comments or something like that.

Covert Gecko: Here.

Dapper Badger: That shows up Here. So, this is, yeah, just some website that has a commenting feature. And maybe this one doesn't have it. This doesn't have the top comments feature. Maybe, yeah. And it's possible that, okay, yeah. So, but we already have some backend API that has implemented this top comments, like, get request. So, that exists, that the developer has already provided this API for getting the top comments. It's just that we have a wrapper that in one case, the frontend code is calling that API for the top comments. And the other one, it's not calling that.

Covert Gecko: Yep.

Dapper Badger: But it's basically the same. We can still interact with the same internal API either way. Right, so it's not like there's two different instances of the API, I guess is what I'm saying. It's just different ways of calling it. Okay, okay, so that makes sense. And I guess there could be, yeah, there could be more complicated ways to do this, but so I guess what I would want to do is have some way, so I'm gonna start trying to draw something here and I guess let's say this is like the app user and, you know, normally the way this would be working is they're interacting with an API. Let's say, yeah, there's various different— there's various different functions in here, but let's just scope it down to— there's function 1 and function 2. And this is the thing that— we want to call this function 1 if the user is in one category and function 2 if the user is in the other category. So I guess—

Covert Gecko: How do you know which category they're in? You're saying like based on frontend or backend or treatment?

Dapper Badger: Yeah, so I'm saying based on treatment. So yeah, this is sort of without our application, but like what we want to do is we want to Essentially have like a routing or like a translation layer that we say that essentially we want to have— I'm gonna make a little bit of extra space here. So—

Covert Gecko: Because essentially your platform is gonna answer for them if it's function one or function, like if you're gonna go down which path. Does that make sense?

Dapper Badger: Right, right, exactly. So I mean, I would imagine here that the user would have basically, you know, we could determine using some kind of like a hash. I'm thinking like almost in a similar way that you could do like hashing for partitioning, that you could basically take some information about the user their email address or something like that. And then it would basically put them into bucket user or like, yeah, this is a layer that basically just determines what bucket a user is in. So maybe I'll just draw a line here. And this is basically a— whoops. I want a text box here. And this is gonna be B down here. And so this is going to be based on, yeah, it's gonna be randomized so that we can split these users up fairly easily. And then depending on which bucket they're in, when they make a certain call from the website— well, when they render the website, they're either going to be calling API function 1 or API function 2. So, it could be kind of like user— if you're in option B, then you call function 1.

Covert Gecko: Mm-hmm.

Dapper Badger: And if you're in bucket A, then maybe you call function 2. Mm-hmm. 2. So, and then obviously this gives a response back to the user and will display it. So there's an arrow going in that direction. So I guess what really needs to happen here though is that I guess what I meant by a wrapper, by having a wrapper, is that, I mean, in general, you could have many different sort of features that could be swapped in. It's not just one function where it could be one or the other. It's really like a mapping of several different components, which depending on which bucket you're in, it's gonna route you to API call 1 or API call 2. And so I think Really, inside this routing layer here, there should be kind of like a mapping of which functions, high-level functions map to which actual implementation in the API. So, like, essentially if it's, you know, comments display or like display comments, let's say. This mapping would say in section A, display comments routes to function 2. And in section B, display comments maps to function 1. And there could be several functions that have that mapping here. And so, it's basically a router. So, yeah. Okay. And then I think there's a similar— I'm kind of focusing on the backend here, but I think a part that I'm not really mentioning here is that this is also obviously important for just the frontend. There's gonna be a lot of frontend features that we want to think about as well here. But I guess I might want to try to run through a complete picture even just with these backend AB testing, like having different implementations of some of the API function calls. And because what we're still not— haven't met all of the functional requirements yet, because what we need is to calculate different metrics here based on which bucket the user is in. So I think this is— so this is going to interact with The API obviously can be tracking various features, and I think, sorry, it can be tracking various metrics and storing them regardless of what the actual backend database is for this application, which is gonna be very application-specific. I think we want to be able to store these metrics in some kind of database that I would say probably because we wanna do a lot of high-level analytics on this, I would recommend using like, and it's sort of like OLAP style interactions as opposed to transactional. I think we probably want a SQL database. And I'm sorry that I don't know how to get the database icon here.

Covert Gecko: That's okay. I actually don't know how to do it either. Every so often somebody pops in with like this incredible shape and I'm like, how did you even do that? Yeah.

Dapper Badger: Well, I'll just make a circle for now. So that's great. So cylinder, top-down view. Great. Uh, so let's just call this, uh, you know, a SQL database. It could be like Amazon RDS or something like that. Um, but this is, uh, metrics database, and that's gonna be SQL. And so I think in this database, what we want to do is, I mean, I may wanna lay out some of this schema here, so the types of things that we would wanna track. So we discussed having, tracking some of these metrics like, The click-through rate, I mean, okay, let's see. There does, I suppose, need to be a separate process that's actually sort of listening, like watching the user interactions here. That's separate. This isn't really triggered by anything in particular, like that the user is, yeah, this is separate from the API that's watching the user. I think there's kind of like a, some process that's actually generating these metrics in real time. Well, it doesn't need to be in real time. I suppose it could also be offline. But I think there's a separate process that is sort of listening to the— well, really, like, the frontend. I think it's— sort of the frontend server that would be able to understand, that would be able to read out how often is the user when they're on a certain page clicking through or how long are they staying on the page, for example. I don't know if there's a best practice there, but so I guess what I'm saying is there might need to be like a separate microservice that is sort of listening to this. So, maybe I'll move this down and I'll say that there's a service that's kind of— it's read-only from the website and the backend that is sort of metrics generation.

Covert Gecko: Like, yeah, you can assume it's kind of black boxy for us, right? Where it's like there's probably already trackers on the website that are pulling metrics.

Dapper Badger: Right, right. Okay, so I don't have—

Covert Gecko: you just need to store the more specific ones about like what treatment the customer is in, as an example.

Dapper Badger: Yeah, that, that makes a lot of sense. Um, but so yeah, there's some service that's doing that. So, but in terms of, um, maybe we can get into the nitty-gritty of what's actually going to be stored here. So We talked about, you know, for each— so let's see. Metrics table. So— and I guess I'm going to assume that there's probably some process where you're like recording individual clicks and things like that, and then that is getting aggregated and sent as like aggregate metrics. I'm going to skip over that intermediate step, and I'm just going to say We're immediately getting aggregated metrics that are on the level of, like, we have sort of a user ID. We have which branch they're in. And then we have time— I'm gonna say it's not timestamp because, again, we're not— I don't think this is, like, an events-level table. I think this is already pre-aggregated just to kind of simplify things. So, I'm gonna say, like, time buckets. So this could be like maybe minutes or— actually, no, because that's probably a little bit overly granular. It could be— I guess I'm— so I'm curious, do we— yeah, this is a question about the functional requirements. Do we care about having these metrics and tracking them over time, or do we wanna just, let's say, well, ever since inception of this branch, here's what the user's average click-through rate has been, or do we want like a chart that we could actually look at of, you know, here's their average click-through rate in these time buckets?

Covert Gecko: Over time, like, I guess, um, for me it's probably a little bit more like on or off in terms of like, okay, for a customer who received A, did it improve the click-through rate over— well, it also depends. Sorry, I'm— that was confusing because A could be control, right? So whichever one is your treatment, did it improve X metric over your control? Yeah. Yep.

Dapper Badger: Okay. Which, that makes sense.

Covert Gecko: Like the Simpson's action things, like since customers are starting to get it, starting to use it.

Dapper Badger: Yep.

Covert Gecko: Yep.

Dapper Badger: That makes sense. Okay. So I think I might not actually even have a time bucket here. I might just go directly to, here's sort of, um, you know, average click-through, uh, and the, you know, time, I forget exactly what this one is called, but like time on site and thinking of some other metrics like, you know, money spent, maybe if there, like it, maybe there's some kind of integration with the payment processor, cuz that would be something we would wanna track. Or, you know, various different metrics here. And then I think it— you mentioned that we want to be able to compare it to the control, so I would say this metrics table should also include information about this user from before the change was instituted, because we don't want to just compare branch A to branch B. We probably want to be able to compare users on the branch that changed with their own data from before. And so I guess I would say that probably this branch ID here should probably— everyone should always be assigned a branch ID. Like, even if you're just not even doing an A/B test, like, you can still— you're still updating some branch ID in this table which could later be referred to. And so I think, yeah, that way you would be able to sort of say the branch that the user was on before the A/B test even started also has a name and we can also look up the aggregate information for that control. So like there's always a branch happening. Everyone is always on a branch. And so, okay, so that just kind of covers how we would compare this to like, to the data from before. Let's say a branch ID is an always incrementing number, and so, you know, you might be on branch 150, not because there's 150 active branches, but just because that includes the full history of your A/B testing over time. And so that's just like an incrementing counter.

Covert Gecko: Okay.

Dapper Badger: So we have this metrics database, and then I guess what another important thing here is we need like someone to be able to read this information for the, the business user, the developer, to obviously to be able to look at this information. So then I think what we need is kind of like a business intelligence dashboard type of thing, or analytics dashboard, let's call it maybe. Mm-hmm. So, like BI dashboard. And this is reading from that SQL database. And it's tracking things like what are the average metrics for one— for each branch. It could have— tracking aggregates, but it could also have statistics. Like it could have some statistical capabilities, like, you know, is it actually statistically significant difference? Like there could be some built-in, you know, basic modeling functionality there that allows the business user to say, you know, is that, you know, can we actually reject the null hypothesis that B is better than A with high confidence, for example. So yeah, I think that would be sort of something that would have to be built by people with statistical expertise. And probably it would have some visualizations as well.

Covert Gecko: Yep.

Dapper Badger: And that would be served to Yeah, to a business user. Maybe I'll just call this, yeah, business user.

Covert Gecko: Okay. Yeah.

Dapper Badger: And okay. Biz user. Okay. So let's see, does that basic setup have, I mean, I think there's a lot of missing components here to cover the missing components that I'm aware of, at least right now, obviously. Scaling, I think— I'm thinking about the scaling thing. You know, there's probably a lot of scaling that's going to be happening inside the original API of that company. If we are really just a wrapper that's routing requests, then, you know, depending on how— what kind of scale we're talking about here, that's a pretty low— that's not a super compute-intensive thing, just sort of translating one request to another and passing it along. But I could imagine if it's like a truly ginormous application that we're wrapping around, that even that routing layer may need to be handling scale well and may need like replication. But I guess for now I'm kind of setting that aside. The other big thing is that I'm kind of really simplifying this into like calling different backend functions and I'm not really paying attention too much to the frontend components and But, so maybe I can talk a bit more about that. But in terms of like the overall framework of you have a user, you have some kind of routing thing that sort of routes you to different, let's call them subsets of the API. So it's not different API, it's not totally different APIs, but kind of different subsets of the API that are available all in one instance. Um, but then eventually it flows through a SQL database and to a business user's dashboard. Um, and I suppose, uh, I'm thinking, well, maybe I'll, uh, just ask that question first. Like, does this overall flow look like it's missing any major components?

Covert Gecko: The only thing I was curious about was in like how you're bucketing your users. Would you be storing anywhere like which user is in which? Uh, yes, that here we talked about a hashing algorithm, um, but curious, like would you have some sort of data store attached to that as well?

Dapper Badger: Yes, uh, sorry, definitely. So I think for, uh, for multiple different things, um, so that's a good point. So, and not just the user's bucket, but Maybe I'll make this a circle. But also some of this information about the mappings because this— that needs to be stored somewhere as well. So, I think this— this is probably more like a key-value store where— I mean, we basically just need to store some— Let's see, we need to store, how do you store? Okay, we need to store which bucket the user is in. User bucket. And then I think there's also gonna have to be, sort of a separate bucket that's gonna store the mappings for the two different versions, for the two different branches here. So, okay. I think for this key-value store, we're gonna have to have like a user ID. So, user mapping. Store, this is gonna be user ID, bucket. Or this, let's call this branch ID. And I mean, that's kind of it, actually. I think. There's almost like a separate, a separate store that is going to be storing more detailed information about the A/B test. Mm-hm. But that could be, 'cause you don't wanna have to store that repeatedly for each user. So, mm-hm. Let's see, maybe I'll write another, thing here that I'll just say, like, branch metadata. Mm-hmm. And so, we need to— this needs to interact. Like, this layer needs to interact with this to sort of determine which functions to call. And it also needs to interact with this. To determine which bucket to send the user. And so when a new user comes in, we can calculate the bucket that they're gonna be in, store it here for later use, and I suppose later on we don't need to recalculate that. We can just sort of read through here. And also then we need to, Anytime that we want to actually route this request, we need to check on the configuration here. Mm-hmm. And that branch metadata, it's gonna be keyed by the branch ID. And then maybe it'll have like a big JSON object that will just tell it which high-level function calls or web— or web pages map to which components or which low-level API calls. In the actual client's API. So, just those mappings basically for each one of the branches. So, yes, there needs to be a place to store that, definitely.

Covert Gecko: Great. I'm gonna pause this right here. We're at the 49.

Dapper Badger: Oh, yes. Wow. Okay.

Covert Gecko: I know.

Dapper Badger: So, that really did go quickly. Uh-huh.

Covert Gecko: Yeah, it does. How are you feeling about it? How do you feel like it went?

Dapper Badger: I think maybe for my first time, it's okay. You know, I think I got some of the— I think I got most of the components for like this very limited version of this kind of functionality. I guess I'm a little disappointed I didn't really get into a lot of the details of like how this is different if you're talking about different frontend components. I didn't quite get into how this could be optimized for like using machine learning. So, I would say the timing was probably— yeah, that was probably the biggest thing. But yeah, how do you think— what do you think were the major things? That I, I guess did well or did poorly.

Covert Gecko: Yep. Yeah. Let's, let's actually start at the top and go through. I'd say like my TL;DR is that you spent more time on the things I expected that you would just based on kind of more of your analytics and ML background. And that's why like I took some notes too to help, help you try to like maximize leaning into where you'll be able to speak most comfortably, but also not miss big topics. So, um, I hope that that's kind of a helpful framing because I think that there were quite a few of like the meatier things that you didn't get to. Um, you started to really get into it more right in like the last 4 minutes, which is why I didn't stop us. Um, and so I think bringing some of those things up earlier is helpful. Okay.

Dapper Badger: So like the key-value store for the users and yeah, what are the other Meteor things that I didn't get into?

Covert Gecko: Yeah, it's a little bit more around like where is the extensibility and scaling that I felt like we could have spent a little bit more time on. Right. We also talked about security.

Dapper Badger: Now, those are like—

Covert Gecko: security isn't necessarily a part of this question per se, so I think it's more fine to ask the question in the beginning of like, can we assume that this is built within the existing ecosystem and handles like authentication? Just like check to make sure. But in terms of like extensibility, there's things like scheduling these things to run Kind of like the multiple treatments that you had mentioned, you know, different access points for different types of users. So for example, software engineers might have different set of APIs that they would use versus like product managers in terms of like the creation versus like the updating or viewing the content. Yeah. And then, splitting out your databases when you think about things like read and write heavy. 'Cause some of these things are read heavy, some of these things are write heavy. We started to get to that right in that last little, the bucket. So I felt good about you understanding that concept, but I was curious if I didn't say anything, would you have brought it up or not?

Dapper Badger: So that was kind of— Yeah, that's a good, definitely a good point. I guess I maybe need to remember if I have any service here, like I need to talk about the storage for that. Service because I kind of had just like implicitly assumed that there's a key-value store here, but I guess anytime I draw a rectangle that's a standalone service, it's got to have some— unless it's an in-memory thing, but obviously this isn't in-memory.

Covert Gecko: So correct, and that's, and that's where it's like you— most people struggle even to come up with the hashing things. You got that so quickly. That I think you were just kind of like, oh, it has everything that I need. But it is helpful to kind of talk about, you know, that even if you're not going into detail, hey, here's, you know, I'm going to have to have a database here just to store treatment information. And, you know, if we scale it to a certain point, like we could consider a caching component. So those are, those are things that like Uh, based on like what you were talking about, I feel like you knew all those things, you just didn't verbalize it.

Dapper Badger: Yeah. Right. I think quickly mentioned read-through caching, but I didn't say much about it at all. Yeah. So I, right. Okay. Thank you. Yeah. Yeah.

Covert Gecko: So in the beginning you asked really good clarification. I purposely make this super vague when I say the question just to kind of see you know, how you can handle ambiguity. I thought you did a really good job of asking a clarifying question. Um, something that can be really helpful though is to frame it around asking for like a user journey or how a user would experience the platform that you can just really then ground the conversation. So, um, that's kind of how I responded to you, um, to kind of help you see anchoring on like the customer experience can really help to make sure you're on the same page with your interviewer. Um, that makes sense.

Dapper Badger: Okay. User journey, kind of like a hint there.

Covert Gecko: Yep. Um, uh, great question in terms of like partitioning by user or session. That's something people usually don't bring up until quite later. So great to see you bring that up right away. Okay. One hint, one hint that I wrote here was to talk about the why behind your questions. You ask me really good questions, but sometimes it can be really helpful to talk about the why.

Dapper Badger: So, um, you know, as an example, right, I, that, that, uh, user versus session thing has a big why behind it for sure. But I didn't mention that.

Covert Gecko: Yeah, exactly. Right, right. And it's like one of those things that's like, I can infer it, but you don't wanna have an interviewer that, that doesn't infer it. Right. And so I think it's actually great to be really explicit and it can be the pros and cons, right? So like, Say you just have no idea even where to go on something. Sometimes it's really, really effective to talk about what something shouldn't be. It gives your brain a chance to catch up. It demonstrates that you have knowledge of good and poor decisions. If you get caught at all, talk about the why behind your questions and then also talk about what something shouldn't be. It would be helpful.

Dapper Badger: Helpful hint. Okay, that makes sense, right? I could, I could have suggested— yeah, made my own opinion about why a session wouldn't make sense because you don't have time to see how the user would actually adapt to the new thing. And correct, right?

Covert Gecko: And again, anchoring it on that customer experience again, because then I think it really anchors you on making sure that you're delivering what they want.

Dapper Badger: Yes. Okay.

Covert Gecko: Which you did, you did a little bit in this one when you're explaining, because you were like, you know, if the user came back, would they want to see the same experience? That's a great example of like how to use the customer within it.

Dapper Badger: Yeah. Okay. That makes sense.

Covert Gecko: Yep. I also find it really helpful at this point to kind of give a high-level agenda or like a walkthrough of how you're going to structure your time. This is also a great place to just kind of mention that kind of shortlist I gave you, which is like, okay, this is like a big question. There's a couple of things I want to make sure that we hit on. I'm going to walk us through the functional and nonfunctional requirements. You know, some things for us to think about is, you know, things like extensibility, security. You can kind of list out that list that I gave you. And then you could say, I'm going to walk through a couple of these things. We'll jump into drawing, but if you have any components you'd like me to spend more time on, let me know so I can make sure to save time for it. That's a helpful way to kind of get your interviewer in the same mindset as you. It also will help you not having to ask the question of, did I get everything, or Am I missing anything? Instead, it's like putting you in the power seat, but also giving me the opportunity to see like, oh, by the way, I really wanna know more about how you're gonna bucket users. So I think that's a helpful reframe of that question you asked.

Dapper Badger: Okay. So right, instead of asking, did I miss anything? Like try to sort of anticipate high-level categories of like what might be missing and write them down. And, and then like, right, and then it's kind of like something that I already mentioned, so it's not like taking me by surprise.

Covert Gecko: Exactly. So like even things like monitoring, we didn't talk about that at all. Super important, right? So you could even just say, you know, we might, we might not have time, but something I want to make sure to mention is that we'd want to have really good monitoring and alarming in our systems to make sure that everything is working as expected.

Dapper Badger: Yep. Yep. Right.

Covert Gecko: So it's, it's nice to mention these things upfront because then your interviewer isn't stuck in the mindset of like, ah, is he gonna talk about monitoring? And then you don't, but instead if you've pre-mentioned it, it kind of makes it be like, oh, he just ran outta time. Yeah.

Dapper Badger: Okay. So maybe after I talk about functional requirements and like non-functional requirements, go into like maybe I was missing just one more kind of high-level list of just like, like high-level components. Yeah, exactly. Okay. And then, right. Gotcha.

Covert Gecko: It also, this is also the strategy in terms of like, oh shoot, I still have 10 minutes left. What am I gonna do with that time? This is the list you can go back to, right? So order these things in your level of comfort. Because what you don't want to do is open a door for them to be like, I want to talk about security, and you're like, oh shoot. So order within kind of your comfort. And the great thing about system design is you steer the whole thing, right? Like you are the driver of this, unlike coding. So I always say try to use that to your advantage.

Dapper Badger: Right. Okay. Yeah. Focus on The things that I'm strongest at.

Covert Gecko: Yep, that makes a lot of sense. Yes, exactly, exactly. Um, so those are just high-level things that would help you avoid missing something. Um, and I think it would, it would help give you some more well-rounded, like, hey, here's all the things we really want to touch on. In terms of like the things that you did draw, um, of course I would have liked to see you spend a little bit more time maybe explicit on things like you know, how would you store that experiment information? But the things that you did share I thought were good, and it didn't surprise me. Again, like, um, I kind of figured you would take more of the BI side of things as we, um, once you do.

Dapper Badger: Um, so when you say like how we would store it, do you mean like the, the specific type of SQL database, or do you mean the part that actually aggregates it, like maybe spending more time on that?

Covert Gecko: Um, I, I was talking a little bit more about like this bubble, right? Oh, I see how you would bucket your users, how you would set up your experiments, um, because there's a lot of extensibility points within this, right? Like some people create an experiment table and a user table and have them separate because your experiment table is going to be a lot more read-heavy, whereas your user table is going to be a lot more write-heavy. And so if you have you know, 1,000 experiments running at the same time, you could then have optionality to create APIs to do like, you know, some sort of like bulk get treatment calls where it's searching what customers would be in, in several different things. So kind of just splitting out that functionality can help. But again, I thought you did a really good job in terms of this like metrics component, which I, I'm not surprised given you're, you're probably a little bit more metrics focused. Yeah. Than a, than a pure software engineer.

Dapper Badger: Mm-hmm. That makes sense. Yeah. Okay. I will definitely keep that in mind. Yeah. Breaking out, I guess. Yeah. The storage and making sure all the different tables and talking about read, reading, read heavy versus write heavy. Yeah. Okay. Yeah.

Covert Gecko: Cause the read versus write heavy shows you're thinking about scale. And that's, that's kind of the root of it, right? Is you just want to make sure that you're, you're thinking about where your platform would fall apart or where it would be easily extensible. Yeah, yeah, absolutely. Yeah. Um, do you feel like the feedback I gave you is something you are able to integrate and feel like you, you know how to incorporate it, or is that something helpful for us to talk through real quick?

Dapper Badger: Um, I, I think so. I think I have some idea of how to incorporate it, but, um, I, I don't know. I guess I mean, I have like a couple of concrete things of try to, try to, you know, some, some concrete takeaways, but yeah, I mean, how to do it in the right amount of time. Um, I mean, if you have any other advice on how to, how to try to incorporate that stuff, um, and still not run out of time, um, like, I don't know, was I talking slowly or was I, should I maybe practice with Just getting faster at actually whiteboarding or, yeah, I guess so.

Covert Gecko: You're, you're a clear and effective communicator, so usually that's a separate problem. So I mean, I was able to follow you. You were talking out loud really well. There was at no point where I was like, what the heck is he doing? So you're doing a great job on all of those things. I think just doing that kind of high level, here are some key buckets for me to be mindful of. Would be effective for you to just at least, you know, kind of knowledge drop a few things. And even if you don't get to it, at least you've mentioned it briefly. And I think that will just help you in your time management component. It'll also give your interviewer some time to be like, oh, hey, I would really love to talk more about the bucket user component. It's a balance. You don't want to give the interviewer too much of like a, hey, where would you like to go? Because then again, you could get caught in an area that you're like, this is not where I am the most comfortable. That's kind of where I feel like it's kind of setting the stage for like, here's my agenda, and then moving allows you to control your narrative a lot more. Um, and allow you to cover a lot more ground.

Dapper Badger: Maybe, okay, maybe if I can summarize one more time, like it sounds like first of all, I mean, as a general heuristic, maybe like don't ask the question of like, am I done? Um, yeah. And instead, like try to leave myself some like loose ends early on so that if I ever finish a chunk, I can sort of, I already have like some stuff queued up I can talk about. Exactly. And right. Okay. And then maybe like, it'll be more like a suggestion. I can be like, okay, so that's done. Next, I can talk about this, but I just wanna give you an opportunity if you want me to talk about something else. But like, I'm making a suggestion instead of just, what do, what do I do? Or are we done? Exactly. Yeah. Okay. That makes sense.

Covert Gecko: I think you cover way more ground faster. It also just like, you just knowledge drop a lot more things faster. It gives you a sense of time management in terms of like, what are the things you truly wanna hit on? So I, I think that that would probably help you the most.

Dapper Badger: Okay. Yeah, that sounds great. And, um, and you mentioned that there's like notes on this as well. Yeah.

Covert Gecko: So I took notes with timestamps. Mm-hmm. Okay. Yep. So I will, I will drop those in our, in our notes here at the end. I will also drop drop those example problems for you. And then I also have just like some really quick, like 5 bullet points of like, here's some high-level things to think about from a behavioral perspective. So I'll drop those in as well.

Dapper Badger: Oh, sure. Yeah, that would be super helpful. Yeah. Awesome.

Covert Gecko: Well, I hope your session later today goes well too. Take some notes, you know, make a little sticky note of like, you know, here's the big 5 categories you want to make sure you hit on, and see if you can implement some of this and see if that helps you with your time management.

Dapper Badger: Yeah, definitely, I will, I will try to do that. I think that's really helpful to have this and then have a follow-up in a little while so I can try to immediately implement some of this stuff. So yeah, yeah, thank you.

Covert Gecko: Exactly, just don't overthink it. I think that's right. That's a challenging part here. Um, and again, I'm, I'm, uh, I'm interested to see how the ML-focused one goes because, um, I could just tell that's kind of where you're more comfortable.

Dapper Badger: So yeah, yeah, it definitely is for sure, but I want to be able to do both. So yeah, exactly. Yeah, this is helpful.

Covert Gecko: Awesome. Well, good luck. Um, if you need anything, don't be shy.

Dapper Badger: Okay, and I could, uh, I could contact you, uh, potentially after this.

Covert Gecko: Is that right, or Yeah, so at the end when you're filling— yeah, so when you fill out feedback for me, um, there'll be a button that says, do you want to unmask? And if we both click it, it will send us an automated email so we have each other's contact information. So, um, that's what I do on my end, so it's totally up to you if that's something you're comfortable with.

Dapper Badger: Okay, yeah, great, I— it definitely is. So, um, yeah, all right, I appreciate that. You bet.

Covert Gecko: All right, good luck.

Dapper Badger: Okay, thank you. All right, bye. Have a good one.

Covert Gecko: You too.