The immediate conclusion is that kids aren’t as easily attentive to words on a page as they are to whatever shiny device is eating their free time up like so many Pringles, so they aren’t able to process longer, more complex sentences.
But that only tells part of the story. Older books needed longer, more descriptive sentences because mass-distributed images and video were not around or not as common. So things had to be described in terms most people could process. That would increase sentence size and, consequently, book length.
Another thing to note is that complex sentence structure doesn’t always compel complex thematic or allegorical structures. Newer. smaller books may have the upperhand in that area. The Media Bistro post quotes something (not sure of the source) that implies that.
Image stolen from Scientopa.