1. Define the token type, including keywords, identifiers and numbers; 2. Create a Lexer structure to read input characters; 3. Implement the NextToken method to identify the token and skip whitespace characters; 4. Test the lexer function through examples. To write a simple lexer in Go, you need to first define the Token structure and related type constants, then create a Lexer structure that saves the input information, then implement the NextToken method to identify different types of tokens and skip whitespace characters, and finally verify the function of lexer through the test code. This process is suitable for beginners to understand and extend.
Writing a simple lexer (lexer) is actually not difficult in Go language. As long as you understand the basic input processing and state control logic, it can be easily implemented. This is not a too complicated compilation principle, but only from a practical perspective, it will take you to write a lexer that can recognize basic keywords, identifiers and numbers step by step.
1. Define the Token type
Before you start writing lexer, you must first clarify which "vocabulary" you want it to recognize. We can define a Token structure to represent each identified element:
type Token struct {
Type string
Value string
}Then define some common token types, such as:
const (
ILLEGAL = "ILLEGAL"
EOF = "EOF"
IDENT = "IDENT" // Identifier INT = "INT" // Integer ASSIGN = "="
PLUS = " "
COMMA = ","
SEMICOLON = ";"
LET = "LET"
)This step is the basis and determines what your lexer can recognize.
2. Create a Lexer structure
Next, create a structure to save the currently read location, characters and other information:
type Lexer struct {
input string
position int // Current character position readPosition int // Next character position to be read ch byte // Current character}The initialization function is as follows:
func New(input string) *Lexer {
l := &Lexer{input: input}
l.readChar()
Return l
} readChar() function is used to read the next character and update the position pointer:
func (l *Lexer) readChar() {
if l.readPosition >= len(l.input) {
l.ch = 0
} else {
l.ch = l.input[l.readPosition]
}
l.position = l.readPosition
l.readPosition
}This part mainly uses byte-byte processing of the input string to lay the foundation for subsequent parsing.
3. Implement NextToken method
This method is responsible for extracting the next token from the input. The general logic is: skip whitespace characters, determine what type of the current character is, and then generate the corresponding token.
func (l *Lexer) NextToken() Token {
var tok Token
l.skipWhitespace()
switch l.ch {
case '=':
tok = newToken(ASSIGN, l.ch)
case ' ':
tok = newToken(PLUS, l.ch)
case ',':
tok = newToken(COMMA, l.ch)
case ';':
tok = newToken(SEMICOLON, l.ch)
case 0:
tok.Type = EOF
tok.Value = ""
default:
if isLetter(l.ch) {
identity := l.readIdentifier()
tok = Token{Type: ident, Value: ident}
} else if isDigit(l.ch) {
tok = Token{Type: INT, Value: l.readNumber()}
} else {
tok = newToken(ILLEGAL, l.ch)
}
}
l.readChar()
Return tok
} Among them, helper functions such as skipWhitespace , isLetter , isDigit , etc. can be written like this:
func (l *Lexer) skipWhitespace() {
for l.ch == ' ' || l.ch == '\t' || l.ch == '\n' || l.ch == '\r' {
l.readChar()
}
}
func isLetter(ch byte) bool {
return 'a' <= ch && ch <= 'z' || 'A' <= ch && ch <= 'Z'
}
func isDigit(ch byte) bool {
return '0' <= ch && ch <= '9'
} There are also implementations of readIdentifier and readNumber :
func (l *Lexer) readIdentifier() string {
position := l.position
for isLetter(l.ch) || isDigit(l.ch) {
l.readChar()
}
return l.input[position:l.position]
}
func (l *Lexer) readNumber() string {
position := l.position
for isDigit(l.ch) {
l.readChar()
}
return l.input[position:l.position]
}These functions can fully identify variable names, integers, symbols, etc.
4. Use examples and tests
You can use this lexer like this:
input := "let x = 5; let y = 10;"
l := New(input)
for {
tok := l.NextToken()
if tok.Type == EOF {
break
}
fmt.Printf("% v\n", tok)
}The output will probably be:
{Type:LET Value:let}
{Type:IDENT Value:x}
{Type:= Value:=}
{Type:INT Value:5}
{Type:; Value:;}
...Of course, this lexer is still very simple and does not deal with more complex situations such as floating point numbers, strings, comments, etc., but it is enough to serve as a basis for learning and expansion.
Basically that's it. The entire process is clear and the structure is simple, which is suitable for beginners to understand and expand. If you plan to continue to be a parser or interpreter, this lexer is a good starting point.
The above is the detailed content of How to write a simple lexer in golang. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Is golang frontend or backend
Jul 08, 2025 am 01:44 AM
Golang is mainly used for back-end development, but it can also play an indirect role in the front-end field. Its design goals focus on high-performance, concurrent processing and system-level programming, and are suitable for building back-end applications such as API servers, microservices, distributed systems, database operations and CLI tools. Although Golang is not the mainstream language for web front-end, it can be compiled into JavaScript through GopherJS, run on WebAssembly through TinyGo, or generate HTML pages with a template engine to participate in front-end development. However, modern front-end development still needs to rely on JavaScript/TypeScript and its ecosystem. Therefore, Golang is more suitable for the technology stack selection with high-performance backend as the core.
How to install Go
Jul 09, 2025 am 02:37 AM
The key to installing Go is to select the correct version, configure environment variables, and verify the installation. 1. Go to the official website to download the installation package of the corresponding system. Windows uses .msi files, macOS uses .pkg files, Linux uses .tar.gz files and unzip them to /usr/local directory; 2. Configure environment variables, edit ~/.bashrc or ~/.zshrc in Linux/macOS to add PATH and GOPATH, and Windows set PATH to Go in the system properties; 3. Use the government command to verify the installation, and run the test program hello.go to confirm that the compilation and execution are normal. PATH settings and loops throughout the process
How to build a GraphQL API in golang
Jul 08, 2025 am 01:03 AM
To build a GraphQLAPI in Go, it is recommended to use the gqlgen library to improve development efficiency. 1. First select the appropriate library, such as gqlgen, which supports automatic code generation based on schema; 2. Then define GraphQLschema, describe the API structure and query portal, such as defining Post types and query methods; 3. Then initialize the project and generate basic code to implement business logic in resolver; 4. Finally, connect GraphQLhandler to HTTPserver and test the API through the built-in Playground. Notes include field naming specifications, error handling, performance optimization and security settings to ensure project maintenance
Go sync.WaitGroup example
Jul 09, 2025 am 01:48 AM
sync.WaitGroup is used to wait for a group of goroutines to complete the task. Its core is to work together through three methods: Add, Done, and Wait. 1.Add(n) Set the number of goroutines to wait; 2.Done() is called at the end of each goroutine, and the count is reduced by one; 3.Wait() blocks the main coroutine until all tasks are completed. When using it, please note: Add should be called outside the goroutine, avoid duplicate Wait, and be sure to ensure that Don is called. It is recommended to use it with defer. It is common in concurrent crawling of web pages, batch data processing and other scenarios, and can effectively control the concurrency process.
Go embed package tutorial
Jul 09, 2025 am 02:46 AM
Using Go's embed package can easily embed static resources into binary, suitable for web services to package HTML, CSS, pictures and other files. 1. Declare the embedded resource to add //go:embed comment before the variable, such as embedding a single file hello.txt; 2. It can be embedded in the entire directory such as static/*, and realize multi-file packaging through embed.FS; 3. It is recommended to switch the disk loading mode through buildtag or environment variables to improve efficiency; 4. Pay attention to path accuracy, file size limitations and read-only characteristics of embedded resources. Rational use of embed can simplify deployment and optimize project structure.
Go for Audio/Video Processing
Jul 20, 2025 am 04:14 AM
The core of audio and video processing lies in understanding the basic process and optimization methods. 1. The basic process includes acquisition, encoding, transmission, decoding and playback, and each link has technical difficulties; 2. Common problems such as audio and video aberration, lag delay, sound noise, blurred picture, etc. can be solved through synchronous adjustment, coding optimization, noise reduction module, parameter adjustment, etc.; 3. It is recommended to use FFmpeg, OpenCV, WebRTC, GStreamer and other tools to achieve functions; 4. In terms of performance management, we should pay attention to hardware acceleration, reasonable setting of resolution frame rates, control concurrency and memory leakage problems. Mastering these key points will help improve development efficiency and user experience.
How to build a web server in Go
Jul 15, 2025 am 03:05 AM
It is not difficult to build a web server written in Go. The core lies in using the net/http package to implement basic services. 1. Use net/http to start the simplest server: register processing functions and listen to ports through a few lines of code; 2. Routing management: Use ServeMux to organize multiple interface paths for easy structured management; 3. Common practices: group routing by functional modules, and use third-party libraries to support complex matching; 4. Static file service: provide HTML, CSS and JS files through http.FileServer; 5. Performance and security: enable HTTPS, limit the size of the request body, and set timeout to improve security and performance. After mastering these key points, it will be easier to expand functionality.
Go select with default case
Jul 14, 2025 am 02:54 AM
The purpose of select plus default is to allow select to perform default behavior when no other branches are ready to avoid program blocking. 1. When receiving data from the channel without blocking, if the channel is empty, it will directly enter the default branch; 2. In combination with time. After or ticker, try to send data regularly. If the channel is full, it will not block and skip; 3. Prevent deadlocks, avoid program stuck when uncertain whether the channel is closed; when using it, please note that the default branch will be executed immediately and cannot be abused, and default and case are mutually exclusive and will not be executed at the same time.
