Backend Development
Golang
How to implement a worker pool pattern in golang for concurrent processing?
How to implement a worker pool pattern in golang for concurrent processing?
Jun 25, 2025 pm 04:57 PM
To implement a worker pool in Go, use goroutines and channels. 1. Define job and result structs to represent tasks and their outcomes. 2. Create a fixed number of workers as goroutines that process jobs from a shared channel. 3. Send jobs to the job channel and collect results from the result channel, ensuring proper synchronization with WaitGroup if needed. 4. Optionally add dynamic scaling and graceful shutdowns using context.Context to control worker lifecycle. This approach enables efficient concurrency management and resource control for independent tasks.
You can implement a worker pool in Go using goroutines and channels. This pattern is useful when you want to manage concurrency and control resource usage, especially for tasks that can be processed independently.
1. Understanding the basic components
A worker pool typically includes:
- A set of workers (goroutines) waiting for jobs
- A job queue (channel) where tasks are sent
- A way to signal when all jobs are done (like a
sync.WaitGroup)
Workers sit idle until a job arrives on the channel. Once they process it, they go back to waiting for more work.
Here’s what a basic structure looks like:
type Job struct {
// your data here
}
type Result struct {
// processed result
}Each job gets processed by a worker, and results can be collected if needed.
2. Setting up workers with goroutines
Start by creating a fixed number of workers — usually based on the number of CPU cores or some system limit you want to enforce.
func worker(id int, jobs <-chan Job, results chan<- Result) {
for job := range jobs {
// process job
fmt.Println("Worker", id, "processing job")
// simulate processing time
time.Sleep(time.Millisecond * 100)
results <- Result{}
}
}Then start multiple workers:
const numWorkers = 3
jobs := make(chan Job, 10)
results := make(chan Result, 10)
for w := 1; w <= numWorkers; w {
go worker(w, jobs, results)
}This gives you a reusable pool that pulls from the same job channel.
3. Sending jobs and collecting results
To use the worker pool, just send jobs into the channel:
for j := 1; j <= 5; j {
jobs <- Job{}
}
close(jobs)And optionally collect the results:
for a := 1; a <= 5; a {
<-results
}If you need to track how many results to expect, wrap this with a sync.WaitGroup.
One thing to note: don’t forget to close the jobs channel after sending all the work. Otherwise, workers might keep waiting indefinitely.
4. Adding dynamic scaling and graceful shutdowns
If you want more flexibility, you can dynamically scale the number of workers based on incoming load.
For graceful shutdowns, consider using a context.Context to notify all workers when it's time to stop:
ctx, cancel := context.WithCancel(context.Background()) // pass ctx into workers
Inside the worker function, check the context’s Done channel before starting new work:
select {
case <-ctx.Done():
return
default:
// continue processing
}This lets you shut down cleanly when the program exits or when the workload completes.
That’s basically how you do a worker pool in Go. It's not complex, but there are a few easy-to-miss details like closing channels and managing shutdown properly.
The above is the detailed content of How to implement a worker pool pattern in golang for concurrent processing?. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
What is the standard project layout for a Go application?
Aug 02, 2025 pm 02:31 PM
The answer is: Go applications do not have a mandatory project layout, but the community generally adopts a standard structure to improve maintainability and scalability. 1.cmd/ stores the program entrance, each subdirectory corresponds to an executable file, such as cmd/myapp/main.go; 2.internal/ stores private code, cannot be imported by external modules, and is used to encapsulate business logic and services; 3.pkg/ stores publicly reusable libraries for importing other projects; 4.api/ optionally stores OpenAPI, Protobuf and other API definition files; 5.config/, scripts/, and web/ store configuration files, scripts and web resources respectively; 6. The root directory contains go.mod and go.sum
How do you read a file line by line in Go?
Aug 02, 2025 am 05:17 AM
Using bufio.Scanner is the most common and efficient method in Go to read files line by line, and is suitable for handling scenarios such as large files, log parsing or configuration files. 1. Open the file using os.Open and make sure to close the file via deferfile.Close(). 2. Create a scanner instance through bufio.NewScanner. 3. Call scanner.Scan() in the for loop to read line by line until false is returned to indicate that the end of the file is reached or an error occurs. 4. Use scanner.Text() to get the current line content (excluding newline characters). 5. Check scanner.Err() after the loop is over to catch possible read errors. This method has memory effect
How do you handle routing in a Go web application?
Aug 02, 2025 am 06:49 AM
Routing in Go applications depends on project complexity. 1. The standard library net/httpServeMux is suitable for simple applications, without external dependencies and is lightweight, but does not support URL parameters and advanced matching; 2. Third-party routers such as Chi provide middleware, path parameters and nested routing, which is suitable for modular design; 3. Gin has excellent performance, built-in JSON processing and rich functions, which is suitable for APIs and microservices. It should be selected based on whether flexibility, performance or functional integration is required. Small projects use standard libraries, medium and large projects recommend Chi or Gin, and finally achieve smooth expansion from simple to complex.
How do you parse command-line flags in Go?
Aug 02, 2025 pm 04:24 PM
Go's flag package can easily parse command line parameters. 1. Use flag.Type() to define type flags such as strings, integers, and booleans; 2. You can parse flags to variables through flag.TypeVar() to avoid pointer operations; 3. After calling flag.Parse(), use flag.Args() to obtain subsequent positional parameters; 4. Implementing the flag.Value interface can support custom types to meet most simple CLI requirements. Complex scenarios can be replaced by spf13/cobra library.
How do you use conditional statements like if-else in Go?
Aug 02, 2025 pm 03:16 PM
The if-else statement in Go does not require brackets but must use curly braces. It supports initializing variables in if to limit scope. The conditions can be judged through the elseif chain, which is often used for error checking. The combination of variable declaration and conditions can improve the simplicity and security of the code.
How do you declare constants in Go?
Aug 02, 2025 pm 04:21 PM
In Go, constants are declared using the const keyword, and the value cannot be changed, and can be of no type or type; 1. A single constant declaration such as constPi=3.14159; 2. Multiple constant declarations in the block are such as const(Pi=3.14159; Language="Go"; IsCool=true); 3. Explicit type constants such as constSecondsInMinuteint=60; 4. Use iota to generate enumeration values, such as const(Sunday=iota;Monday;Tuesday) will assign values 0, 1, and 2 in sequence, and iota can be used for expressions such as bit operations; constants must determine the value at compile time,
What does the go run command do?
Aug 03, 2025 am 03:49 AM
gorun is a command for quickly compiling and executing Go programs. 1. It completes compilation and running in one step, generates temporary executable files and deletes them after the program is finished; 2. It is suitable for independent programs containing main functions, which are easy to develop and test; 3. It supports multi-file operation, and can be executed through gorun*.go or lists all files; 4. It automatically processes dependencies and uses the module system to parse external packages; 5. It is not suitable for libraries or packages, and does not generate persistent binary files. Therefore, it is suitable for rapid testing during scripts, learning and frequent modifications. It is an efficient and concise way of running.
How to connect to a SQL database in Go?
Aug 03, 2025 am 09:31 AM
To connect to SQL databases in Go, you need to use the database/sql package and a specific database driver. 1. Import database/sql packages and drivers (such as github.com/go-sql-driver/mysql), note that underscores before the drivers indicate that they are only used for initialization; 2. Use sql.Open("mysql","user:password@tcp(localhost:3306)/dbname") to create a database handle, and call db.Ping() to verify the connection; 3. Use db.Query() to execute query, and db.Exec() to execute
