Use json.Decoder to implement streaming JSON decoding to process large files. 1. Use json.NewDecoder to replace json.Unmarshal, and read data block by block through io.Reader; 2. Define Go structures to extract information in advance, such as parsing meta fields containing versions and timestamps; 3. Skip unnecessary data, use decoder.Token() to control the parsing process or decoder.Decode(&unused) to discard useless content; 4. Efficiently process large arrays, first detect the Delim'[' tag, then parse and process each element item by item to avoid loading the entire array into memory.
Sometimes, when you're working with large JSON data in Go, loading the entire JSON into memory isn't efficient or even possible. That's where streaming JSON decoding comes in handy. It lets you process JSON piece by piece without holding the whole thing in memory — great for big files or long HTTP responses.
Use json.Decoder instead of json.Unmarshal
The key to streaming JSON decoding is using json.Decoder , not json.Unmarshal . The Decoder reads from an io.Reader , like a file or HTTP response body, and processes JSON incrementally.
For example, if you're reading from a file:
file, _ := os.Open("big-data.json")
defer file.Close()
decoder := json.NewDecoder(file)Or if it's an HTTP request:
resp, _ := http.Get("http://example.com/data.json")
defer resp.Body.Close()
decoder := json.NewDecoder(resp.Body)Once you have a decoder set up, you can start reading tokens one at a time or decode known structures as they come.
Decode known parts as structs, skip unknowns
If your JSON has predictable sections (like headers or metadata), define Go structs for them and decode those directly.
Say you have JSON that starts like this:
{
"meta": {
"version": 1,
"timestamp": "2023-01-01T00:00:00Z"
},
"items": [ ... potentially huge list ... ]
} You can define a struct for meta :
type Meta struct {
Version int `json:"version"`
Timestamp time.Time `json:"timestamp"`
}Then read it early on:
var meta Meta decoder.Decode(&meta)
This way, you extract useful info upfront and move on to stream the rest without having to parse everything at once.
Skip fields or values you don't need
Sometimes there are parts of the JSON you don't care about, but they're still big. You can skip them entirely using Decode with a blank interface or even better, use Skip() if you're walking through tokens manually.
Here's how you might do it:
for {
tok, err := decoder.Token()
if err == io.EOF {
break
}
// process or skip tokens here
} Using Token() gives you more control — especially useful if you want to find specific keys and ignore the rest.
Alternatively, if you know a section won't be used, just:
var unused interface{}
decoder.Decode(&unused)That tells the decoder to consume and discard whatever value comes next — object, array, string, etc.
Handle arrays efficiently
If the JSON contains a large array, especially under a known field like "items" , you can decode each item one at a time.
First, look for the Delim [ token:
for decoder.More() {
tok, _ := decoder.Token()
if delim, ok := tok.(json.Delim); ok && delim == '[' {
break
}
}Then loop through each item:
for decoder.More() {
var item ItemStruct
decoder.Decode(&item)
// process item
}This avoids loading the entire array into memory and lets you process items as they arrive.
Streaming JSON decoding in Go is pretty straightforward once you get used to json.Decoder . Just remember to work with io.Reader , decode what you need, and skip or stream the rest. It's not magic, but it makes handling big JSON much more manageable.
Basically that's it.
The above is the detailed content of How to stream JSON decoding in Go. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
What are the implications of Go's static linking by default?
Jun 19, 2025 am 01:08 AM
Go compiles the program into a standalone binary by default, the main reason is static linking. 1. Simpler deployment: no additional installation of dependency libraries, can be run directly across Linux distributions; 2. Larger binary size: Including all dependencies causes file size to increase, but can be optimized through building flags or compression tools; 3. Higher predictability and security: avoid risks brought about by changes in external library versions and enhance stability; 4. Limited operation flexibility: cannot hot update of shared libraries, and recompile and deployment are required to fix dependency vulnerabilities. These features make Go suitable for CLI tools, microservices and other scenarios, but trade-offs are needed in environments where storage is restricted or relies on centralized management.
How do I create a buffered channel in Go? (e.g., make(chan int, 10))
Jun 20, 2025 am 01:07 AM
To create a buffer channel in Go, just specify the capacity parameters in the make function. The buffer channel allows the sending operation to temporarily store data when there is no receiver, as long as the specified capacity is not exceeded. For example, ch:=make(chanint,10) creates a buffer channel that can store up to 10 integer values; unlike unbuffered channels, data will not be blocked immediately when sending, but the data will be temporarily stored in the buffer until it is taken away by the receiver; when using it, please note: 1. The capacity setting should be reasonable to avoid memory waste or frequent blocking; 2. The buffer needs to prevent memory problems from being accumulated indefinitely in the buffer; 3. The signal can be passed by the chanstruct{} type to save resources; common scenarios include controlling the number of concurrency, producer-consumer models and differentiation
How does Go ensure memory safety without manual memory management like in C?
Jun 19, 2025 am 01:11 AM
Goensuresmemorysafetywithoutmanualmanagementthroughautomaticgarbagecollection,nopointerarithmetic,safeconcurrency,andruntimechecks.First,Go’sgarbagecollectorautomaticallyreclaimsunusedmemory,preventingleaksanddanglingpointers.Second,itdisallowspointe
How can you use Go for system programming tasks?
Jun 19, 2025 am 01:10 AM
Go is ideal for system programming because it combines the performance of compiled languages ??such as C with the ease of use and security of modern languages. 1. In terms of file and directory operations, Go's os package supports creation, deletion, renaming and checking whether files and directories exist. Use os.ReadFile to read the entire file in one line of code, which is suitable for writing backup scripts or log processing tools; 2. In terms of process management, the exec.Command function of the os/exec package can execute external commands, capture output, set environment variables, redirect input and output flows, and control process life cycles, which are suitable for automation tools and deployment scripts; 3. In terms of network and concurrency, the net package supports TCP/UDP programming, DNS query and original sets.
What are functional options patterns in Go, and when are they useful for constructor design?
Jun 14, 2025 am 12:21 AM
FunctionaloptionsinGoareadesignpatternusedtocreateflexibleandmaintainableconstructorsforstructswithmanyoptionalparameters.Insteadofusinglongparameterlistsorconstructoroverloads,thispatternpassesfunctionsthatmodifythestruct'sconfiguration.Thefunctions
How do I call a method on a struct instance in Go?
Jun 24, 2025 pm 03:17 PM
In Go language, calling a structure method requires first defining the structure and the method that binds the receiver, and accessing it using a point number. After defining the structure Rectangle, the method can be declared through the value receiver or the pointer receiver; 1. Use the value receiver such as func(rRectangle)Area()int and directly call it through rect.Area(); 2. If you need to modify the structure, use the pointer receiver such as func(r*Rectangle)SetWidth(...), and Go will automatically handle the conversion of pointers and values; 3. When embedding the structure, the method of embedded structure will be improved, and it can be called directly through the outer structure; 4. Go does not need to force use getter/setter,
What are interfaces in Go, and how do I define them?
Jun 22, 2025 pm 03:41 PM
In Go, an interface is a type that defines behavior without specifying implementation. An interface consists of method signatures, and any type that implements these methods automatically satisfy the interface. For example, if you define a Speaker interface that contains the Speak() method, all types that implement the method can be considered Speaker. Interfaces are suitable for writing common functions, abstract implementation details, and using mock objects in testing. Defining an interface uses the interface keyword and lists method signatures, without explicitly declaring the type to implement the interface. Common use cases include logs, formatting, abstractions of different databases or services, and notification systems. For example, both Dog and Robot types can implement Speak methods and pass them to the same Anno
How do I use the io package to work with input and output streams in Go?
Jun 20, 2025 am 11:25 AM
TheGoiopackageprovidesinterfaceslikeReaderandWritertohandleI/Ooperationsuniformlyacrosssources.1.io.Reader'sReadmethodenablesreadingfromvarioussourcessuchasfilesorHTTPresponses.2.io.Writer'sWritemethodfacilitateswritingtodestinationslikestandardoutpu
