Read about an LLM internals. How it's not reasoning, how it's not doing math at all, and how you are fooled and how you can fool it.
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
If you prefer a quick TL;DR:
https://www.youtube.com/watch?v=-wzOetb-D3w
