Small models are better at cost-to-recall than large models like Mythos for vulnerability research

TL;DR: If a large model finds a 0-day with 90% probability, and a small model with 50% probability, but the small model costs 10x less, it is better to use the small model. We compared the cost and recall of various models in finding real, recent zero-days and found that for most applications, smaller models run repeatedly can significantly outperform larger frontier models on cost-to-recall. Disclaimer: I'm involved with Hacktron, the company that produced this research. This is a factual presentation of our benchmarks, which we hope the community can use to make informed decisions about models like Mythos. submitted by /u/EliteRaids [link] [comments]

from hacking: security in practice https://ift.tt/QfYIiu7