That chart is ideally how it works, and how it usually does work. Unfortunately these can vary significantly by manufacturer, especially between smaller companies. I universal way of determining yarn size is wraps per inch, just wrap the yarn around a ruler. There are lots of charts online for it
It sounds like people are making excuses like goldilocks, saying too thick or too thin or too rough or too soft. Usually the pattern will tell you what yarn and needle size is suitable for that pattern.
I think it matters more how you think it feels rather than if it says it's soft.
Maybe you could try using some of the yarns that say they're for kids/ babies?