How do I use Laravel's chunk to avoid running out of memory?

Vote count:

0

I am pulling around 100k records from temporary tables, doing some slight modification on the data, downloading photos, and then putting the fields I need to keep in a "master" table. This is quickly causing my app to crash as it runs out of memory.

I read the very brief docs on using chunk() with Laravel's eloquent ORM but do not know how to even begin to implement it in my class.

Here is what I am currently doing:

    public function fire()
{
    // Turn off query logging
    DB::connection()->disableQueryLog();

    $feeds = RetsFeed::where('active','=',1)->get();
    foreach ($feeds as $feed)
    {

        $class = "TempListing{$feed->board}";

        $listings = $class::orderBy('MatrixModifiedDT','desc')->get();

        $listings->each(function($listing) use ($feed) {
            ListingMigrator::migrateListing($listing,$feed);
            echo "Feed: $feed->board\r\n";
            echo "SubcondoName: $listing->SubCondoName\r\n";
            echo "Development: $listing->Development\r\n";
            echo "\r\n";
        });
    }

}

Each feed (or datasource) is dumped into a temp table in a different chore. That works fine. Then, I grab all of hte listings out of one table (which is around 30k on average) and run my ListingMigrator method.

Where do I put the chunk in this example? Would it replace the line:

$listings = $class::orderBy('MatrixModifiedDT','desc')->get();

I don't fully understand the closure in the eloquent docs. This is all they have to say about it and here's the code example from the Laravel site:

    User::chunk(200, function($users)
{
    foreach ($users as $user)
    {
        //
    }
});

asked 2 mins ago

Chris Farrugia

121

How do I use Laravel's chunk to avoid running out of memory?