LLMs are solving MCAT, the bar test, SAT etc like they’re nothing. At this point their performance is super human. However they’ll often trip on super simple common sense questions, they’ll struggle with creative thinking.
Is this literally proof that standard tests are not a good measure of intelligence?
All standardized test is how well you prepared for that particular standardized test, doesn’t matter if it is the SAT, MCAT, or Leetcode. You aren’t suppose to think on the spot for these tests, you are suppose regurgitate everything you have rehearsed for weeks and months during the test.
And unthinking regurgitation is what LLMs do better than anything else.
I would argue that some code test questions can be solved spontaneously, but they are limited to easy to some early medium questions, or patterns that are common enough.
I guess this is more common in non FANG companies that don’t have to filter out candidates just because of the sheer number alone.
As someone that didn’t really have good coaching on the SAT, I 100% agree. I kinda fucked it up, but at 17, I wasn’t really used to studying for things outside of school and my parents didn’t get me into any study classes
For GRE though, I studied my ass off… got top 96 percentile scores.
Also went through the leetcode grind. Bombed the first job search I ever did and then later aced the hell out of it after studying really hard.
These tests are all about how diligently you studied and your study technique.