How to Efficiently Select Random Rows from Large PostgreSQL Tables?
Jan 21, 2025 am 05:26 AM
PostgreSQL random row selection method
Traditional random row selection methods are inefficient and slow when dealing with large tables containing millions or even billions of records. Two common methods are:
-
Use
random()to filter:select * from table where random() < 0.001;
-
Use
order by random()andlimit:select * from table order by random() limit 1000;
However, due to the need for a full table scan or sorting, these methods are not the best choice for tables with a large number of rows and will cause performance bottlenecks.
Optimization methods for large tables
For the following types of tables, consider the following optimization method, which is significantly faster:
- Numeric ID columns with small or medium gaps (indexed for faster lookups)
- No or minimal write operations during selection
Query:
WITH params AS (
SELECT 1 AS min_id, -- 可選：自定義最小ID起始值
5100000 AS id_span -- 近似ID范圍（最大ID - 最小ID + 緩沖）
)
SELECT *
FROM (
SELECT DISTINCT 1 + trunc(random() * p.id_span)::integer AS id
FROM params p, generate_series(1, 1100) g
GROUP BY 1
) r
INNER JOIN big ON r.id = big.id
LIMIT 1000;
How it works:
-
ID range estimate:
- If not known exactly, query the table to estimate the minimum, maximum, and total span (max - min) of the ID column.
-
Random ID generation:
- Generate a different set of random numbers within the estimated ID range.
-
Redundancy and duplication elimination:
- Group the generated numbers to remove duplicates, reducing the possibility of selecting missing rows or already selected rows.
-
Table joins and restrictions:
- Join the random numbers with the actual table using the ID column (must be indexed). This efficient join retrieves the corresponding data for the selected row.
- Finally, apply a limit to retrieve the required number of rows.
Why it’s fast:
-
Minimal index usage:
- The query only performs an index scan on the ID column, which is much faster than a full table scan or sort operation.
-
Optimized random number generation:
- The generated random numbers are distributed over the estimated ID range, minimizing the possibility of missing or overlapping rows.
-
Redundancy and duplication elimination:
- Grouping the generated numbers ensures that only distinct rows are selected, reducing the need for additional filtering or joining to eliminate duplicates.
Other options:
-
Recursive CTE to handle gaps:
- For tables with gaps in the ID sequence, add an additional CTE to handle these gaps.
-
Function wrappers for reuse:
- Define a function that takes limit and gap percentage as parameters, allowing easy configuration and reuse with different tables.
-
Universal functions for any table:
- Create a generic function that accepts any table with integer columns as a parameter.
-
Materialize views for speed:
- Consider creating a materialized view based on an optimized query for faster retrieval of (quasi) randomly selected rows.
-
TABLE SAMPLEin PostgreSQL 9.5:- Leverage PostgreSQL's "
TABLE SAMPLE SYSTEM" feature to implement a faster but less random row sampling method, ensuring an accurate number of rows is returned. However, keep in mind that the sample may not be completely random due to clustering effects.
- Leverage PostgreSQL's "
The above is the detailed content of How to Efficiently Select Random Rows from Large PostgreSQL Tables?. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Establishing secure remote connections to a MySQL server
Jul 04, 2025 am 01:44 AM
TosecurelyconnecttoaremoteMySQLserver,useSSHtunneling,configureMySQLforremoteaccess,setfirewallrules,andconsiderSSLencryption.First,establishanSSHtunnelwithssh-L3307:localhost:3306user@remote-server-Nandconnectviamysql-h127.0.0.1-P3307.Second,editMyS
How to add the MySQL bin directory to the system PATH
Jul 01, 2025 am 01:39 AM
To add MySQL's bin directory to the system PATH, it needs to be configured according to the different operating systems. 1. Windows system: Find the bin folder in the MySQL installation directory (the default path is usually C:\ProgramFiles\MySQL\MySQLServerX.X\bin), right-click "This Computer" → "Properties" → "Advanced System Settings" → "Environment Variables", select Path in "System Variables" and edit it, add the MySQLbin path, save it and restart the command prompt and enter mysql--version verification; 2.macOS and Linux systems: Bash users edit ~/.bashrc or ~/.bash_
What are the transaction isolation levels in MySQL, and which is the default?
Jun 23, 2025 pm 03:05 PM
MySQL's default transaction isolation level is RepeatableRead, which prevents dirty reads and non-repeatable reads through MVCC and gap locks, and avoids phantom reading in most cases; other major levels include read uncommitted (ReadUncommitted), allowing dirty reads but the fastest performance, 1. Read Committed (ReadCommitted) ensures that the submitted data is read but may encounter non-repeatable reads and phantom readings, 2. RepeatableRead default level ensures that multiple reads within the transaction are consistent, 3. Serialization (Serializable) the highest level, prevents other transactions from modifying data through locks, ensuring data integrity but sacrificing performance;
Where does mysql workbench save connection information
Jun 26, 2025 am 05:23 AM
MySQLWorkbench stores connection information in the system configuration file. The specific path varies according to the operating system: 1. It is located in %APPDATA%\MySQL\Workbench\connections.xml in Windows system; 2. It is located in ~/Library/ApplicationSupport/MySQL/Workbench/connections.xml in macOS system; 3. It is usually located in ~/.mysql/workbench/connections.xml in Linux system or ~/.local/share/data/MySQL/Wor
Performing logical backups using mysqldump in MySQL
Jul 06, 2025 am 02:55 AM
mysqldump is a common tool for performing logical backups of MySQL databases. It generates SQL files containing CREATE and INSERT statements to rebuild the database. 1. It does not back up the original file, but converts the database structure and content into portable SQL commands; 2. It is suitable for small databases or selective recovery, and is not suitable for fast recovery of TB-level data; 3. Common options include --single-transaction, --databases, --all-databases, --routines, etc.; 4. Use mysql command to import during recovery, and can turn off foreign key checks to improve speed; 5. It is recommended to test backup regularly, use compression, and automatic adjustment.
Analyzing the MySQL Slow Query Log to Find Performance Bottlenecks
Jul 04, 2025 am 02:46 AM
Turn on MySQL slow query logs and analyze locationable performance issues. 1. Edit the configuration file or dynamically set slow_query_log and long_query_time; 2. The log contains key fields such as Query_time, Lock_time, Rows_examined to assist in judging efficiency bottlenecks; 3. Use mysqldumpslow or pt-query-digest tools to efficiently analyze logs; 4. Optimization suggestions include adding indexes, avoiding SELECT*, splitting complex queries, etc. For example, adding an index to user_id can significantly reduce the number of scanned rows and improve query efficiency.
Handling NULL Values in MySQL Columns and Queries
Jul 05, 2025 am 02:46 AM
When handling NULL values ??in MySQL, please note: 1. When designing the table, the key fields are set to NOTNULL, and optional fields are allowed NULL; 2. ISNULL or ISNOTNULL must be used with = or !=; 3. IFNULL or COALESCE functions can be used to replace the display default values; 4. Be cautious when using NULL values ??directly when inserting or updating, and pay attention to the data source and ORM framework processing methods. NULL represents an unknown value and does not equal any value, including itself. Therefore, be careful when querying, counting, and connecting tables to avoid missing data or logical errors. Rational use of functions and constraints can effectively reduce interference caused by NULL.
Resetting the root password for MySQL server
Jul 03, 2025 am 02:32 AM
To reset the root password of MySQL, please follow the following steps: 1. Stop the MySQL server, use sudosystemctlstopmysql or sudosystemctlstopmysqld; 2. Start MySQL in --skip-grant-tables mode, execute sudomysqld-skip-grant-tables&; 3. Log in to MySQL and execute the corresponding SQL command to modify the password according to the version, such as FLUSHPRIVILEGES;ALTERUSER'root'@'localhost'IDENTIFIEDBY'your_new
