Table of Contents

[Definitive Guide] Operating Scheduled Jobs & Batches with Laravel Scheduler × Queues — From Reliable Execution and Monitoring to Accessible Progress Notifications

What you’ll learn (key takeaways)

Designing safe operations for scheduled tasks with Laravel’s Scheduler (app/Console/Kernel.php)
Implementing & monitoring large-scale batches by combining queues (ShouldQueue / retries / delays)
Idempotency, duplicate-run prevention, time zones, and maintenance windows
Accessibility considerations for deliverables (report generation, imports, notifications): screen reader, focus, and state representations not dependent on color
An operational checklist, rollback strategies on failure, and CI/CD integration

Who is this for? (Who benefits?)

Laravel beginners to intermediates who need to run scheduled/nightly batches in production
Tech leads at agencies or SaaS companies who want stable operational patterns with queues and scheduler
CS/Ops teams who want to streamline user notifications and failure detection to reduce tickets
Accessibility leads & QA who want progress/complete/failure messaging that “anyone can understand”

1. Introduction: Why “Scheduler × Queue”?

In business systems and SaaS, you need processes that “run at a fixed time reliably,” such as nightly batches, hourly aggregations, and morning email deliveries. Laravel’s Scheduler lets you register just one OS cron entry while controlling triggers flexibly in the app. For heavy workloads or external API integrations, it’s standard to pair with the job queue. This enables:

Centralized control in application code for timing, frequency, and exception handling
Each process as an async job with retries and worker distribution
Easier design of failure alerts and user-facing notifications
Future scalability by breaking tasks down and composing them in series/parallel

This article carefully explains how to design, implement, and operate scheduled tasks that “run safely, quietly, and reliably,” including accessibility perspectives.

2. Basic Wiring: One cron line, scheduler managed in the app

2.1 cron configuration (server side)

* * * * * php /path/to/artisan schedule:run >> /dev/null 2>&1

Running schedule:run every minute will execute only the schedules defined in app/Console/Kernel.php when needed.
If you put the same cron on multiple servers, use locks (see below) or a “leader-only execution” strategy.

2.2 Basics of `app/Console/Kernel.php`

// app/Console/Kernel.php
namespace App\Console;

use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;

class Kernel extends ConsoleKernel
{
    protected function schedule(Schedule $schedule): void
    {
        // Generate a sales report every day at 03:00
        $schedule->command('report:daily-sales')
            ->dailyAt('03:00')
            ->timezone('Asia/Tokyo')
            ->withoutOverlapping(30)        // Skip if a previous run is still executing (30-min lock)
            ->onOneServer()                 // Single execution in multi-server setups
            ->emailOutputOnFailure('ops@example.com');
    }

    protected function commands(): void
    {
        $this->load(__DIR__.'/Commands');
    }
}

Key points:

Explicitly set timezone('Asia/Tokyo') to avoid time differences and DST issues.
Use withoutOverlapping() to prevent double starts.
onOneServer() is a safeguard for single-node execution in distributed environments.
emailOutputOnFailure() detects failures and notifies ops (other channels are fine too).

3. Artisan Commands and Queues: Always async for heavy tasks

3.1 Create a command (thin orchestration)

php artisan make:command GenerateDailySalesReport

// app/Console/Commands/GenerateDailySalesReport.php
namespace App\Console\Commands;

use Illuminate\Console\Command;
use App\Jobs\BuildSalesReport; // See below

class GenerateDailySalesReport extends Command
{
    protected $signature = 'report:daily-sales {--date=}';
    protected $description = 'Generate daily sales report';

    public function handle(): int
    {
        $date = $this->option('date') ?? now()->subDay()->toDateString();
        BuildSalesReport::dispatch($date)->onQueue('reports');

        $this->info("Dispatched BuildSalesReport for {$date}");
        return Command::SUCCESS;
    }
}

The command should focus on orchestration only; move heavy lifting into a job.
Allow passing a target date option to make re-runs easy later.

3.2 Create a job (business logic core)

php artisan make:job BuildSalesReport

// app/Jobs/BuildSalesReport.php
namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Queue\SerializesModels;
use App\Services\Reports\SalesReportService;

class BuildSalesReport implements ShouldQueue
{
    use InteractsWithQueue, Queueable, SerializesModels;

    public int $tries = 3;               // Automatic retry count
    public int $timeout = 3600;          // Timeout 1h

    public function __construct(public string $date) {}

    public function middleware(): array
    {
        return [
            new RateLimited('reports'),   // Name for rate limiting calls to external APIs, etc.
        ];
    }

    public function handle(SalesReportService $svc): void
    {
        // Idempotency check (skip if already generated)
        if ($svc->exists($this->date)) {
            return;
        }

        // Fetch large datasets by chunks, then generate CSV/Excel/PDF
        $svc->build($this->date);

        // Completion notifications (Email/Web/Slack, etc.)
        $svc->notifyCompleted($this->date);
    }

    public function failed(\Throwable $e): void
    {
        // Failure notification & auxiliary info
        // e.g., $svc->notifyFailed($this->date, $e->getMessage());
    }
}

Design points:

Async with ShouldQueue. Explicitly set tries and timeout.
Add rate limiting or unique controls via middleware().
Put idempotency (e.g., exists check) first so duplicate runs are safe.
Use failed() to notify cause and remedy on errors.

4. Duplicate Runs and Idempotency: Prevent accidents in advance

4.1 Scheduler locks and unique jobs

Add withoutOverlapping(minutes) to schedules.
On the queue side, prevent duplicate enqueues with a unique key (e.g., Redis lock).

// Simple unique enqueue example (in a service layer, etc.)
$lockKey = "job:build-sales:{$date}";
if (cache()->lock($lockKey, 600)->get()) {
    BuildSalesReport::dispatch($date);
    // Optionally release() the lock on job start/end
}

4.2 Idempotent design

Save deliverables (report files, etc.) with a deterministic path/ID; if it exists, skip or overwrite.
Ensure external APIs can be re-fetched with the same results using paging/cursors.
For DB updates, use transactions + upserts to withstand partial failures.

5. Scaling: Split large tasks into small ones

5.1 Chunks and sequential processing

// Aggregating a large number of rows
Order::whereDate('created_at', $this->date)
    ->orderBy('id')
    ->chunk(5_000, function ($chunk) use ($writer) {
        foreach ($chunk as $order) {
            $writer->appendRow($order);
        }
    });

Keep memory usage low and improve stability for long-running jobs.
Default to streaming outputs (write to files progressively) for aggregation results.

5.2 Parallelization and dependencies

Split many jobs by day/store/category, then dispatch.
If there are dependencies, control parent → child order or separate lanes by queue name.
Stage queues (e.g., “report generation → email sending”) to make monitoring easier.

6. Monitoring and Visibility: Detect failures quickly, recover quietly

6.1 Monitoring job states

Aggregate failed jobs (in the failed_jobs table) every morning and notify.
Use Supervisor, etc., for auto-restart and log preservation of workers.
Instrument queue backlog (latency/count) and alert by thresholds.

6.2 Scheduler health checks

Send a heartbeat that “the scheduler is alive” every day at 00:05.
Alert on days when it’s not running.
Run schedule:list / schedule:test periodically to prevent configuration drift.

7. Maintenance windows and time zones

Avoid heavy jobs during business quiet hours.
Use skip(function() { ... }) to branch by business days/holidays.
Design schedules to match partner API rate-limit windows.
For user notifications, consider the recipient’s time zone.

$schedule->command('notify:daily')
    ->dailyAt('08:00')
    ->timezone('Asia/Tokyo')
    ->skip(function () {
        return today('Asia/Tokyo')->isWeekend();
    });

8. Typical use cases and sample implementations

8.1 CSV import (nightly batch)

Detect files uploaded to S3 → split and enqueue
In each job validate → reflect to DB (accumulate validation errors in a separate table)
After completion, email an aggregate report and an error list

// app/Jobs/ImportChunk.php
public function handle(ImportService $svc): void
{
    $rows = $svc->read($this->path, $this->offset, $this->limit);
    foreach ($rows as $row) {
        $dto = $svc->validate($row);   // Collect failures and notify later
        $svc->upsert($dto);            // Idempotent
    }
}

8.2 External API aggregation (hourly)

Throttle queues with RateLimited middleware to match API rate limits
On intermittent failures, auto-retry; after a certain count, move to a quarantine queue and let ops decide

9. Accessibility for user-facing progress & results

9.1 Progress dashboard (Web)

{{-- Read out progress summaries via live region --}}
<div id="status" aria-live="polite" role="status" class="mb-3">
  Today’s report generation is 60% complete (3/5 steps)
</div>

<ul class="space-y-2">
  <li>
    <span class="inline-flex items-center gap-2">
      <span aria-hidden="true">⏳</span>
      Data extraction
    </span>
    <span class="sr-only">In progress</span>
  </li>
  <li>
    <span class="inline-flex items-center gap-2 text-green-700">
      <span aria-hidden="true">✔</span>
      Aggregation
    </span>
    <span class="sr-only">Completed</span>
  </li>
  <li>
    <span class="inline-flex items-center gap-2 text-gray-700">
      <span aria-hidden="true">…</span>
      File output
    </span>
  </li>
</ul>

Combine color with icons + text to represent states redundantly.
Use aria-live="polite" for screen-reader announcements on progress updates.
Ensure keyboard operability; implement actions as links/buttons.

9.2 Email copy for completion/failure

Provide both HTML and plain-text versions.
Keep subject lines short in the order of “datetime + content + result.”
On failure, clearly state whether retry is possible, point of contact, and key error points.
Don’t rely on images or colors; ensure the body text alone conveys meaning.

10. Error handling and rollback

Use transactions to avoid partial success.
For external APIs, adopt circuit breakers and exponential backoff.
Classify failure types (invalid data / transient / permanent) and vary retry policies.
For chained jobs, cancel children on parent failure and unify notifications.

11. Security and permissions

Issue least-privilege env vars/credentials for batches.
Distribute deliverables via signed URLs or temporary auth tokens safely.
Don’t log personal or sensitive data (mask it).

12. Horizon/Supervisor operations (overview)

With Redis queues, visualization and throttling settings are easy.
Estimate worker count by processing time × arrival latency, then scale gradually.
Always log job ID / target date / category as search keys.

13. Quality assurance: Test strategy

13.1 Unit/feature tests (sync execution)

public function test_report_command_dispatches_job(): void
{
    Queue::fake();
    $this->artisan('report:daily-sales --date=2025-08-31')
        ->assertExitCode(0);

    Queue::assertPushed(\App\Jobs\BuildSalesReport::class, function ($job) {
        return $job->date === '2025-08-31';
    });
}

Validate job dispatches with Queue::fake().
Keep the service layer pure and cover it with table-driven tests.

13.2 E2E (small-scale real runs)

Use small input data to actually generate files → verify integrity.
Mock failure paths (API 403/429, network issues) to confirm retries and notifications.

14. Checklist (for handouts)

Schedule design

[ ] Explicit timezone(); add withoutOverlapping() / onOneServer()
[ ] skip() conditions for business days / quiet hours
[ ] Failure notifications & heartbeat

Job design

[ ] Configure ShouldQueue / tries / timeout / RateLimited
[ ] Idempotency (existence checks, deterministic paths, upserts)
[ ] Chunk/stream to respect memory limits
[ ] In failed(), notify cause and next steps

Scale/splitting

[ ] Dispatch by day/store/category
[ ] Separate lanes with queue names; control dependency order

Monitoring/operations

[ ] Aggregate failed jobs & backlog alerts
[ ] Monitor workers & auto-restart via Supervisor/Horizon
[ ] Log search keys (job ID/date/category)

Accessibility

[ ] Use role="status" / aria-live for progress/result screens
[ ] Represent success/failure with more than color (icons/text)
[ ] Always include a plain-text email version

Security

[ ] Least-privilege credentials
[ ] Deliverables via signed URLs
[ ] Mask sensitive data in logs

15. Concrete templates (finished examples, excerpts)

15.1 Console Kernel

protected function schedule(Schedule $schedule): void
{
    // Heartbeat
    $schedule->call(fn() => \Log::info('scheduler:alive'))
        ->dailyAt('00:05')
        ->timezone('Asia/Tokyo')
        ->onOneServer();

    // Daily report
    $schedule->command('report:daily-sales')
        ->dailyAt('03:00')
        ->timezone('Asia/Tokyo')
        ->withoutOverlapping(60)
        ->onOneServer()
        ->emailOutputOnFailure('ops@example.com');
}

15.2 Signed downloads for deliverables

// routes/web.php
Route::get('/reports/{date}', [ReportController::class, 'show'])
    ->middleware('signed')
    ->name('reports.show');

// Generation
$url = URL::temporarySignedRoute('reports.show', now()->addHours(24), ['date' => $date]);

15.3 Simple polling for progress updates

async function poll() {
  const res = await fetch('/api/report/progress?date=2025-08-31', {headers:{Accept:'application/json'}});
  if (res.ok) {
    const { percent, stepText } = await res.json();
    const el = document.getElementById('status');
    el.textContent = `Progress ${percent}%: ${stepText}`;
  }
  setTimeout(poll, 15000);
}
poll();

16. Common pitfalls and how to avoid them

Writing heavy logic synchronously in the command → Always extract to a job to gain retries/monitoring.
Results change on every run → Ensure idempotency (existence checks, deterministic keys, upserts).
“Completion emails only” → You’ll miss failures. Add failure notifications and a heartbeat.
Progress shown by color alone → Hard to distinguish states. Use icons + wording too.
No time zone specified → Runs at unexpected times. Explicitly set timezone().
Reading huge CSVs at once → Memory pressure. Use chunking and streamed output.

17. Tips to embed into team operations

Standardize command/job/service layers so new ones start from templates.
Improve scanability with naming conventions (Build*, Import*, Notify*).
Standardize output locations & naming for deliverables (reports/YYYYMM/DD/sales_{date}.csv).
Consolidate ops contact for failure notifications—avoid personal inboxes.
Aggregate daily success/failure counts on dashboards to spot anomalies early.

18. Wrap-up: Iterate safely in small steps, scale quietly

Encode time control in the scheduler and stay safe with withoutOverlapping() and onOneServer().
Let jobs do the work with retries, rate limits, and idempotency for resilience.
For large volumes, use chunking and parallelization to push throughput, and make backlog visible with metrics.
Present results with accessibility in mind from the start: screen reader support, non-color cues, and plain-text emails so anyone can understand.
Failure isn’t evil; it’s a signal. Use failure notifications and heartbeats to catch issues early and recover quietly.

Use the templates in this article as a foundation to build scheduled processes that are “quiet, reliable, and kind” for your team. I’m rooting for you.