Lexers/parsers, when written together, generally interface each other either as "parser calls lexer.PeekToken()/lexer.AdvanceToken() until it's done parsing, or

Lexers/parsers, when written together, generally interface each other either as "parser calls lexer.PeekToken()/lexer.AdvanceToken() until it's done parsing, or lexer returns EOF", or, rarer, "lexer calls parser.ConsumeToken(tok) until there is no more input left, and calls parser.Done()". IIRC, SQLite uses the second approach to parse SQL. At no point does any of these schemes require the lexer to read the whole file into memory, and chop it in one go into a huge list of ready-to-be-consumed tokens.