Simplified way of processing large data using chunk() in laravel
As a laravel developer, by large data I mean collection of 1000 or more rows from a single data model that is, a database table; specifically on a MySQL / MariaDB server.
Problems with processing large data ?
There may be situations when a single table holds millions of records and you want to migrate those records to another table after processing. As far as MySQL is concerned here in this context, PHP has some limitations in terms of memory allocation and script execution time to process those rows.
PHP takes data into its allocated memory on server and then process further, also it automatically lends some extra bytes if it thinks it would be sufficient for the script to run and exit successfully. All these actions are handled automatically until the records are less 1000 but once it crosses this limit server resources become deficient of memory.
In short, server stops responding to other requests and many script is exited abruptly, which we call it as a memory leakage. In return, we get a Fatal error: Allowed memory size of <some_number> bytes exhausted (tried to allocate <some_number> bytes)
On the other hand, when the calculation during data processing is way too much complex then script would exit abruptly because each script has a definite execution time allocated.
Now you may wonder, that we can manually change these settings in php.ini file but I do not recommend to mess with any configuration file on the production server unless it is the last resort.
What do we mean by data chunks ?
What if we divide whole collection into chunks, let’s say a collection of 25K rows and we divide them in chunks of 500 rows at a single instance, fair enough ? Yes, because now rows are 500 only at given time. The PHP script will take only 500 rows out of 25K rows. Let’s see some examples related to arrays first.
Array chunks in PHP
In PHP, there is an array helper function, array_chunk().
<?php
$arr_a = [1, 2, 3, ....., 5000];
foreach(array_chunk($arr_a, 500) as $x){
// array_chunk() will divide $arr_a into smaller array as [[1, 2, 3..., 500],[501, 502, .... , 1000] and so one till 5000]
//do some stuff with $x;
}
Laravel Eloquent chunks
In Laravel, collection can be chunked using chunk() method for example,
<?php
Namespace App\Http\Controller;
Use User;
Public function processUsers()
{
User::chunk(500, function($chunked_collection){
//do some stuff with $chunked_collection
});
}
Best practices to free unused memory in PHP
PHP provides many ways to flush unused memory so that the next line of code can be accommodated into available memory.
Unset a php variable
Using unset() method, we can free up the memory a variable is holding up, for example,
<?php
$a = 500;
unset($a);
var_dump($a); // prints null
?>
Nullify a php variable
Another way to free up memory is to reset the variable’s value to null explicitly, but it is slower than unset(), I do not recommend it.
Close database connection in the end manually
While using core PHP, always make sure to close database connection after processing has been done on data. For example:
<?php
$con=mysqli_connect("localhost","my_user","my_password","my_db");
// ....some PHP code...
mysqli_close($con);
?>
Ebooks available now
You can download this article’s PDF eBooks for offline reading from below:
