Review questions whether language models replace specialised architectures

Researcher Kiyan Rezaee published a preprint reviewing whether language-based models can replace specialised machine-learning architectures for structured data. The review covers 159 papers across nine modalities, considering representation and computation alongside prediction accuracy.

It identifies settings where language-based approaches are competitive, including tasks with very few examples, symbolic problems and knowledge graphs annotated with text. But when studies directly evaluate the handling of structure, the review finds no evidence of general architectural replacement. Systems often reintroduce the missing structure through graph modules, specialised attention or other components.

The paper also distinguishes improved performance from proof that scaling will eliminate the remaining gap. That broader claim remains untested in the surveyed work.

The practical distinction is between replacing a user interface and replacing the machinery underneath it: a language-driven system may still depend on specialised models to do its work.