Special Research Paper on “Applications of Data Science and Artificial Intelligence in Economic and Environmental Geology”

Split Viewer

Econ. Environ. Geol. 2024; 57(6): 665-680

Published online December 31, 2024

https://doi.org/10.9719/EEG.2024.57.6.665

© THE KOREAN SOCIETY OF ECONOMIC AND ENVIRONMENTAL GEOLOGY

Full Waveform Inversion Using the Hypergradient Descent Method

Jun Hyeon Jo, Wansoo Ha*

Department of Energy Resources Engineering, Pukyong National University, Busan 48513, Republic of Korea

Correspondence to : *wansooha@pknu.ac.kr

Received: August 29, 2024; Revised: November 29, 2024; Accepted: December 13, 2024

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided original work is properly cited.

Abstract

Optimizing step length or learning rate is crucial for efficient gradient-based inversions, including seismic full waveform inversions and deep learning. Hypergradient descent methods, initially proposed for deep learning, update hyperparameters using gradient descent techniques. We applied the hypergradient descent method to update the step length in full waveform inversion. While this approach still requires selecting an appropriate learning rate for hypergradient descent, it eliminates the need to manually tune and schedule the step length in full waveform inversion. We implemented the hypergradient descent method with the Adam optimizer to invert seismic data and compared the results to those obtained using a line search method. Numerical examples demonstrated that the hypergradient descent method accelerated full waveform inversion and produced results comparable to those from the conventional line search method.

Keywords full waveform inversion, deep learning, hypergradient descent method, step length, learning rate

Article

Special Research Paper on “Applications of Data Science and Artificial Intelligence in Economic and Environmental Geology”

Econ. Environ. Geol. 2024; 57(6): 665-680

Published online December 31, 2024 https://doi.org/10.9719/EEG.2024.57.6.665

Copyright © THE KOREAN SOCIETY OF ECONOMIC AND ENVIRONMENTAL GEOLOGY.

Full Waveform Inversion Using the Hypergradient Descent Method

Jun Hyeon Jo, Wansoo Ha*

Department of Energy Resources Engineering, Pukyong National University, Busan 48513, Republic of Korea

Correspondence to:*wansooha@pknu.ac.kr

Received: August 29, 2024; Revised: November 29, 2024; Accepted: December 13, 2024

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided original work is properly cited.

Abstract

Optimizing step length or learning rate is crucial for efficient gradient-based inversions, including seismic full waveform inversions and deep learning. Hypergradient descent methods, initially proposed for deep learning, update hyperparameters using gradient descent techniques. We applied the hypergradient descent method to update the step length in full waveform inversion. While this approach still requires selecting an appropriate learning rate for hypergradient descent, it eliminates the need to manually tune and schedule the step length in full waveform inversion. We implemented the hypergradient descent method with the Adam optimizer to invert seismic data and compared the results to those obtained using a line search method. Numerical examples demonstrated that the hypergradient descent method accelerated full waveform inversion and produced results comparable to those from the conventional line search method.

Keywords full waveform inversion, deep learning, hypergradient descent method, step length, learning rate

    Fig 1.

    Figure 1.The Marmousi velocity model (Versteeg 1994).
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 2.

    Figure 2.(a) A shot gather without noise and (b) a shot gather with random noise.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 3.

    Figure 3.(a) The initial velocity model for inversions, (b) the inversion result using the line search method, and (c) that using the hypergradient descent method after 500 iterations.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 4.

    Figure 4.(a) The loss curves in the logarithmic scale and (b) the loss curves as a function of the number of wave propagation modeling. The line search method requires three wave propagation models per iteration, while the hypergradient descent method requires only two. (c) Variation in the step length of the inversion process.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 5.

    Figure 5.(a) The inversion result of noisy data using the line search method and (b) that using the hypergradient descent method.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 6.

    Figure 6.(a) The normalized loss curves and (b) the variation in the step length for noisy data.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 7.

    Figure 7.(a) The loss curves and (b) the step lengths of the inversions using the line search method with different D in (11).
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 8.

    Figure 8.(a) The loss curves and (b) the step lengths of the inversions using the hypergradient descent method with different learning rates, η in (13), when α0 = 0.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 9.

    Figure 9.(a) The loss curves and (b) the step lengths of the inversions using the hypergradient descent method with different initial step lengths, α0 in (13), when η = 1.0×10-5.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 10.

    Figure 10.The loss curves using the Adam optimizer with fixed step lengths. Note that the oscillations are exaggerated in the logarithmic scale.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 11.

    Figure 11.(a) The loss curves and (b) the step lengths of the inversions using a mini-batch scheme when D = 400 for the line search method and η = 1.0×10-5 for the hypergradient descent method.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 12.

    Figure 12.(a) The loss curves and (b) the step lengths of the inversions using a mini-batch scheme when D = 2000 for the line search method and η = 2.0×10-6 for the hypergradient descent method.
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Fig 13.

    Figure 13.(a) The inversion result of mini-batch optimization using the line search method with D = 2000 and (b) that using the hypergradient descent method with η = 2.0×10-6 .
    Economic and Environmental Geology 2024; 57: 665-680https://doi.org/10.9719/EEG.2024.57.6.665

    Table 1 . Final loss values, model MAE, and computation time for full-batch waveform inversions. The model MAE of the initial velocity model is 0.3108.

    MethodDataFinal LossModel MAECalculation Time (500 iterations)
    Line searchClean1.1632e-80.225615316.3 s
    Hypergradient descent1.0369e-80.23229761.7 s
    Line searchNoisy5.0160e-70.232615364.3 s
    Hypergradient descent5.0585e-70.24199864.4 s

    Table 2 . Hyperparameters, final loss values, model MAE, and computation time for mini-batch waveform inversions.

    MethodHyperparameterFinal LossModel MAECalculation Time (500 epochs)
    Line searchD = 4001.9423e-80.185815457.2 s
    Hypergradient descentη =1.0 ×10-51.1855e-80.210310789.3 s
    Line searchD = 20001.0887e-80.186515642.0 s
    Hypergradient descentη = 2.0 × 10-61.0784e-80.228911002.1 s

    Table 3 . Number of iterations and computation times for full-batch waveform inversions using relative loss criteria.

    MethodDataTarget LossNumber of IterationsCalculation Time (500 iterations)
    Line searchClean1% of the initial loss35310842.2 s
    Hypergradient descent3486808.5 s
    Line searchNoisy30% of the initial loss1755380.8 s
    Hypergradient descent2344612.4 s

    KSEEG
    Dec 31, 2024 Vol.57 No.6, pp. 665~835

    Stats or Metrics

    Share this article on

    • kakao talk
    • line

    Related articles in KSEEG

    Economic and Environmental Geology

    pISSN 1225-7281
    eISSN 2288-7962
    qr-code Download