Screenshot of this question was making the rounds last week. But this article covers testing against all the well-known models out there.

Also includes outtakes on the ‘reasoning’ models.

  • turboSnail@piefed.europe.pub
    link
    fedilink
    English
    arrow-up
    5
    arrow-down
    2
    ·
    5 days ago

    Well, they are language models after all. They have data on language, not real life. When you go beyond language as a training data, you can expect better results. In the meantime, these kinds of problems aren’t going anywhere.

    • VoterFrog@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      5 days ago

      Why act like this is an intractable problem? Several of the models succeeded 100% of the time. That is the problem “going somewhere.” There’s clearly a difference in the ability to handle these problems in a SOTA models compared to others.

    • dil@lemmy.zip
      link
      fedilink
      English
      arrow-up
      2
      ·
      5 days ago

      Language model means you communictae through natural language I thought