I nearly fell off my chair when my fabulous friend and colleague, the legendary Jon Krohn, invited me to be interviewed on his podcast. This isn’t your typical friend’s-garage-podcast situation – SuperDataScience just happens to be the most listened to Data Science podcast on the planet! Thankfully, the topic was LLM Engineering and selecting LLMs – a subject I can (and frequently do) ramble on about with unbridled enthusiasm for hours on end, let alone my allotted ninety minutes.

To the SuperDataScience listeners who’ve found their way here – you’re either gluttons for punishment or genuinely curious, and I’m super happy either way! After surviving my unofficial TED talk on benchmarks and leaderboards, you’ve somehow come back for more. As promised, I’ve curated the essential LLM resources that Jon and I discussed on the show.

Somewhat self-centered LLM Resources

• My top-rated 8 week immersive course on LLM Engineering, the new exploding profession where Software Engineering meets Data Science
• My LLM Engineering Live Events on the O’Reilly platform
• My LLM game called Outsmart in which models compete to outwit each other
• A project to create a simulation of myself from my 240k text message history
• I love connecting with people on LinkedIn – did Jon say that I’m a “LION”?!

Breaking news…

Read the astounding results of the new OpenAI O3 reasoning model on the ARC-AGI-1 evaluation, and enjoy the astonished-but-ever-snarky reactions on Hacker News, such as:

Let me know when OpenAI can wrap Christmas gifts. Then I’ll be interested.

Leaderboards!

• Hugging Face Open LLM
• Hugging Face Big Code
• Hugging Face LLM-Perf – switch to the ‘Find Your Best Model’ tab
• All Hugging Face leaderboards – medical, Portuguese and more
• Vellum.ai Leaderboard – includes BBHard, also Cost & Context Window comparison
• SEAL specialist leaderboards from Scale.ai
• LiveBench – challenging benchmarks with minimal dataset contamination
• LM Arena (formerly known as LMSYS Arena) and contribute your votes here

Common tools and frameworks used in LLM engineering

1. Hugging Face – needs to introduction! The ubiquitous hub for models, datasets, leaderboards and even applications, and the authors of essential open source frameworks including the pioneering transformers library
2. LangChain – open source framework that provides abstractions connecting multiple LLM operations under a simple API, and glue code to build AI applications with ease; also LangGraph for Agentic AI
3. Gradio – a ridiculously simple UI framework that lets you create prototype UIs in one line of code, no frontend experience needed
   • Alternatives include Streamlit, Dash and Mesop from Google
4. Weights & Biases – tooling to analyze and visualize during training
5. Google Colab – write, evaluate and share notebooks remotely on a box in the Google Cloud
   • Amazon SageMaker is a broader alternative that includes Notebooks
6. Modal.com the serverless AI platform
7. Crewai.com the Agentic AI platform

Parting words

As one final resource to leave you with, here is a recent blog post on Agentic architectures from Anthropic that truly resonated with me – particularly the emphasis on keeping it simple and avoiding unnecessary abstractions.

Thank you once again to the incomparable Jon for having me on the show, and to all of you for making it to the bitter end! Did I mention that I’m just slightly partial to connecting on LinkedIn?? For me, connections are like leaderboards: you can never have too many.