Roblox Engineer Reveals How to Ace FAANG ML Interviews—From Junior to Staff+

Part 1 was published last December and has proved to be useful to so many reader and a lot of the candidates reached out to get more pointers on how to approach ML interviews. Hence this article.

Machine learning system design round is increasingly becoming one of the rounds of assessment at top FAANG companies for ML engineers. The caveat is that ML System design interviews are still not formalised or haven’t become standardised across the industry. This is both a good thing and a bad thing. It’s a good thing because, as a candidate if you are prepared well, you can really shine through this round compared to other candidates. It’s a bad thing because, depending on the company and the interviewer this round can be a hit or miss.

I have interviewed at 10+ Tier1 and Tier2 companies and secured offers for Staff ML levels from Google, Uber, Doordash, Nextdoor, Square, Glean, Roblox and a few others. I have also interviewed at smaller companies ahead of my Tier 1 onsite interviews. I am currently a Principle Machine learning engineer at Roblox. I plan to share tips on what helped me in securing onsite offers based on my experience attending over 7+ ML design interviews and taking over 10+ mock interviews for colleagues and candidates.

If you haven’t read my previous article , please do take a look https://medium.com/p/78a2882a05c5. I’ve also shared some study resources here. This article is less about giving you interview questions or prep materials, but more about giving you a sense of criteria you have to watch out for when you interview at a certain level. ML System design interviews are quite crucial at calibrating your actual IC level.

Most ML System design interview books in the market don’t give a complete picture of how a candidate should approach this round.

The industrial standards are still being set and from my perspective, books on ML System design interviews (Alex Xu, Khang Pham, Master ML) still don’t precisely capture expectations from interviewers especially from FAANG companies where building production grade ML systems processing requests for Billions of users over millions of entities (Think ads ranking) requires a very niche skill set of ML that overlaps ever so lightly with traditional Machine learning. So, if you are a candidate in the Job market, do not go completely by what you study from these books. My main pet peeves about the books are : Books seem to cater both to ML Modeling engineers as well ML Infra engineers with Khang Pham’s book more leaning towards Infra engineers.

• The outline/Structure of the book caters more to L4 engineers than Senior/Staff/Senior staff engineers. In my opinion, the content touches the surface of each of the topics like Feature engineering, Online metrics, offline metrics and doesn’t go into depths that you are typically required to present at interviews.
• There’s a lot of emphasis on traditional ML concepts which practically do not work in a production system of Meta/Google scale. So you might end up with talking points that’s outdates or legacy.
• Does that mean you should ignore the books — Certainly no. I myself have skimmed both the books to prepare for the interviews. Take parts of the book and don’t treat them as sufficient to your ML Prep.

In this article, I plan to provide some suggestions on how levels are calibrated and how an L4, L5, L6 engineer may approach the same system design question.

Say you have a question like “Design a Youtube recommendation system” -> There’s a typical structure that you could follow:

In my previous article I’ve touched upon how you could prepare for an ML design system with a template. This is a simple template that’s pretty common for all levels, however the depth of the answers will help differentiate a candidate at L5 vs. L6.

Here’s how an L4 Engineer might answer the question

• An L4 engineer would typically be expected to complete project level tasks in one component (i.e Build a feature set for X, y,z , Train an offline model for existing dataset). We expect L4 engineers to know the basic building blocks. Typically there are 5 parts to the ML Design question. As an L4, you could spend the first 10 mins giving an overview of the system and just taking two parts to deep dive. Typically Feature engineering and Model building are L4 favourites and also presents a depth in modeling.

• 10 mins - feature engineering -> How do you encode categorical features, how do you log features, embedding features

• 10 mins - Modeling -> How do you model this problem in logistic regression vs. Decision trees. Speak about Cross entropy, pros and cons of LR and GBDT. Difference between GBDT and XGBoost. How do you parameter tune them?

  Common L4 Red flags

  • Not presenting an overview of the problem. Unable to decompose the ML design into a fundamental classification approach.
  • Switching from one section to another in a haphazard way. For e.g I interviewed a candidate while I was at Meta, who started with objective function (Prob. of click), started talking about features and then came back and changed objective function to a Multimodal problem.

Here’s how an L5 Engineer might answer the question

• An L5 Engineer is typically expected to go deep in one component and also cross collaborate with two or more components outside their scope. (i.e Build the ML library for two tower model and deploy the first version, Build the A/B experiment interleaving platform, Build the contextual ranking modeling system etc) L5 Engineer should show the depth that an L4 engineer shows — typically within the first 15 mins and move on to discuss more novel concepts.
• Expect to give an overview of the entire system — starting from objective function, training data generation , model building and Deployment.

Expect to give trade offs for each option

• Objective function — compare between pairwise learning vs. pointwise learning.

• Training data generation — Give 2–3 options of how you could generate training data (for.e.g you could generate sessionized training data, speak about weighting negative samples or upweighting long tail samples. Trade off on cases like what happens if there’s popularity bias on one video/game. How do you handle new users?

• Model building — Contrast among different options but don’t spend too much time here. An L5 Engineer would know that Models are just a part of the equation as opposed to a lot of junior engineers who get fixated on Models.

• Deployment — Talking about A/B experimentation (interleaving vs non-interleaving), User exposure, Minimum detectable effect itself can take upto 10 mins.

• Without being asked explicitly already touch on offline-online metric skew. So you offline trained a model and put it in AB Experiment and you notice the metrics are not as expected? What do you do?

• Feature engineering — Usually you could just skip this or just spend a minute here. This really is a place where L4 shines and too much time here doesn’t exhibit L5 signs.

• Online deployment/Objective function and Metrics is typically where an L5 engineer would shine through.

  Common L5 red flags

  • Spending too much time on model architecture or feature engineering.
  • Not asking clarifying questions on corpus size, qps and the need for real time predictions -
  • A candidate jumping ahead and proposing a deep neural network when your corpus size is <5000 is a red flag.
  • Typically candidates totally brushing aside easier algos like Logistic regression and directly recommending Deep cross network is also a big red flag. DCN’s are rarely used outside of FAANG and you would come across as a buzzword liker.
  • Forgetting to spend considerable amount of time on what happens during productionisation. This really is the bottleneck of production grade ML systems.
  • We expect an L5 engineer to have a holistic understanding of the entire system. Candidates fitting the same set of Multi stage system (candidate generation -> ranking -> re-ranking) for questions like “Recommend grocery replacement when search item is not present” is an overkill and a red flag-> This was a questions asked in Instacart and I know a candidate who made the solution too complex while both the corpus and qps was low enough for a single decision tree model.

Here’s how an L6 & above engineer might answer the question

L6 engineers typically are tech lead for the entire domain and have cross-org impact. At bigger companies like Meta, 3–4 L6 folks lead the CTR ads prediction for a particular surface (marketplace/IG), 3–4 L6 folks lead the notifications ML system for say IG Story etc. At smaller startups like Roblox/Instacart/Doordash — these are engineers who would build the entire building block.

“Design the first ads prediction system for Meal delivery startup”

“Design and implement the first ranking model for homepage personalization”

To exhibit the depth that is required for such complex problems, an L6 engineers needs to have higher degree of taking the birds eye view, speaking about the broader picture, always always talking about tradeoffs and noticing patterns

Expect to cover what an L5 would cover in a succinct way in the first 15 mins and then get to more advanced topics

• Providing the tradeoffs for each section is a must. L6 candidates can typically skip easier sections like Feature engineer. An L6 candidate will most likely not spend time on modeling intricacies.
• Touch on nuanced concepts in a detailed manner (e.g Popularity bias, Introducing diversity in ranking, Can we apply a repetition penalty, Label leakage/novelty effect)

Call out best practices

• If I have a team of 5 engineers — would I build a simple logistic regression model or a multi stage recommendation system.
  – Downsides of multistage recommendation -> Seasoned candidates who’ve worked with recommendation system know that jumping on the bandwagon can shoot yourself in the foot 3 years down the line.
• What are some practices you would employ in your team to ensure you build a scalable high Roi team.
  – This often requires 2+ years worth of vision into not just the best and great of ML tech but also requires a bit of production experience to keep away from bad design patterns and choices.
• Modeling for privacy is such an important topic and shines an L6 candidate very well — how do you build with non-pii data
  – Ways to improve dev-efficiency — L6 engineer is the one who constantly thinks about Dev-efficiency.If something is repeatedly done in a team more than thrice — Platformization that is a best practice. For e.g Everytime a new ads surface is added on IG, how do you bootstrap ranking — Going in and retraining all models for the new surface (adding feats, training data, model parity etc) is a laborious process.
• L6+ Engineers constantly think of building out new scope for the team, while taking a long view and reducing overlaps with other teams. Its important to internalize traits of a successful E6 engineer to be able to bring those as talking points in your ML design interview.

I hope this article was useful. I write more articles on ML Design case studies on my website www.helloml.co. I also provide free mock interviews for folks who are currently in the job market and here’s my booking link https://prepfully.com/coach/7VNNX