As the leading data and evaluation partner for frontier AI companies, Scale is dedicated to advancing the evaluation and benchmarking of large language models (LLMs). We are building industry-leading LLM leaderboards, setting new standards for model performance assessment. Our mission is to develop rigorous, scalable, and fair evaluation methodologies to drive the next generation of AI capabilities.
We are seeking Research Scientists and Research Engineers with expertise in LLM evaluation. You will play a key role in developing and implementing novel evaluation methodologies, metrics, and benchmarks to assess the capabilities and limitations of our cutting-edge LLMs. We encourage collaborations within the industry and academia, and support the publication of research findings. Successful candidates will partner with top foundation model labs, providing both technical and strategic input on the development of the next generation of generative AI models.
You will:
- Design and develop novel evaluation benchmarks for large language models, covering areas such as coding, instruction following, factuality, robustness, and fairness.
- Conduct research on the effectiveness and limitations of existing LLM evaluation techniques.
- Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols.
- Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
- Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives.
Ideally you’d have:
- Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
- Strong background in deep learning and LLMs, with experience in model evaluation.
- Familiarity with benchmarking tools and datasets for LLM evaluation.
- Hands-on experience large-scale model training and deployment.
- Excellent written and verbal communication skills.
- Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
- Previous experience in a customer facing role.
