github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-11-06 21:58:01 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

f2fs-for-7.4-rc1

Browse Source 
In this round, key enhancements focus on introducing a dedicated metadata
 cache infrastructure and dynamically parameterizing on-disk block layouts
 and geometry macros. Moving meta, node, and compress blocks out of the VFS
 page cache removes the page == block size assumption for metadata, which
 along with dynamic block layout descriptions paves the way for large folio
 and page != block support in the future. We also opt in to multigrain
 timestamps, add bio entry-count throttling for asynchronous write_end_io
 completion, and expose the resizable_tail feature via sysfs.
 
 Alongside these features, critical bug fixes resolve race conditions across
 compression, mmap write faults, checkpoint/shutdown, and filesystem freeze,
 prevent write bio splitting and GC holes on sequential zoned devices, and
 strengthen on-disk sanity checks for SIT/NAT bitmaps, read-only curseg
 entries, and device aliasing. Additional stability patches address online
 resize, roll-forward recovery, atomic file operations, quota handling, and
 dirty inode syncing livelocks.
 
 Enhancement:
  - implement dedicated metadata cache for meta, node, and compress blocks
    to decouple metadata from page cache for large folio and page != block
    support
  - dynamically describe SIT, NAT, orphan, dentry, node, and xattr block
    layouts and parameterize block/sector/node conversion macros
  - opt in to multigrain timestamps
  - introduce max_atc_write_bio_entry_cnt sysfs node
  - expose resizable_tail feature via sysfs
  - avoid unnecessary reads in f2fs_update_meta_block()
 
 Bug fix:
  - compress: force out-place update for all writes, allow pinning empty
    compressed files, and fix truncate/writeback and cache population races
  - zone: avoid bio split and GC target block grab failure on sequential zones
  - quota: convert inline quota files on quotaon, fix flush failure during
    freeze, and restore empty string mount options
  - atomic/mmap: disallow mmap write and data-modifying fallocate on atomic
    files, avoid truncate_inode_pages_final() on live inodes, and fix races in
    f2fs_vm_page_mkwrite()
  - resize/alias: refresh pinned allocation boundary and flush SIT journal
    during resize, and drop pending discards before reserving device alias
  - sanity check: bound SIT/NAT bitmaps by checkpoint pack, check all
    cur_*_segno[] on readonly images, and validate multi-device on device alias
  - recovery/cp: release removed xattr node in roll-forward recovery, invalidate
    curseg on change_curseg() failure, and return -EIO for lost inode block
    after cp_error
  - fix livelock in syncing dirty inodes and preserve APPEND/UPDATE ino entries
  - use bio_in_atomic() in f2fs_{read,write}_end_io()
  - fix 32-bit shift truncation in mount option bitmasks and defragment range
 
 And, the series includes a number of minor bug fixes.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEE00UqedjCtOrGVvQiQBSofoJIUNIFAmrfjdgACgkQQBSofoJI
 UNKNnQ/9Huhqc4w1mMmhMLRgXAdlkPOt7uDt4Pj4iOJXmIqWab3TMj/6Kc2vL0Hz
 ns6JAKsZ2tMLCBmbnvUdYc1Cbn2SiyM+Ewu23IbnqY0J9BwqPGRrexEDUcivkoui
 yYalPskGnujT2U4WicYLfe6AqZWbvkF7X8MbYwtRn6nHIy1OvUsRwjyVodK7a+wy
 P9oqPT/PqabrV2BV3B3GAcYndsI+g1mliAAVkmhATV6ZQB5A+YpnyGqBC9SWauJl
 tEYnTyDskZegs1iISXJQ5BBaT2XKJyhPpjF3SR0U9RE3ZBYV9P3CN3RtiF0Uto8d
 7G1UQ+IYc8Bz1BPY7rN5vM8V8UJ9DE8i3dEOcwC+RqqTA3eKnknpcnc0s+6Lrc5k
 sj7qI/AwJuARO/IU094+o+P0qKF4vedPmSagXMzMNYbgmtYD5oNHbPjZD/IXApQk
 Csnc2QAccrwodQWQC+IUvqbdokvbMhMDWq+oRzS+zztE/DBP/moWn1P2Br3Qhh+f
 906KZZrS7yW+KjaNYuatL918mvdHxbFxsJ5n0OfWuqxuvlxrhI3towOpBrWJyB8n
 sXpP4khvKuBTCPmRCR/PMP5M5QrjFNeDcXv5C0E8clP0iItrQsuic40OLdgC44QS
 8dvIXdUm73DHTeZtl//ZZKI4fTo0/JFVzfsLJMEfFPNAhh+R4LE=
 =/CUz
 -----END PGP SIGNATURE-----

Merge tag 'f2fs-for-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs

Pull f2fs updates from Jaegeuk Kim:
 "In this round, key enhancements focus on introducing a dedicated
  metadata cache infrastructure and dynamically parameterizing on-disk
  block layouts and geometry macros.

  Moving meta, node, and compress blocks out of the VFS page cache
  removes the page == block size assumption for metadata, which along
  with dynamic block layout descriptions paves the way for large folio
  and page != block support in the future.

  We also opt in to multigrain timestamps, add bio entry-count
  throttling for asynchronous write_end_io completion, and expose the
  resizable_tail feature via sysfs.

  Alongside these features, critical bug fixes resolve race conditions
  across compression, mmap write faults, checkpoint/shutdown, and
  filesystem freeze, prevent write bio splitting and GC holes on
  sequential zoned devices, and strengthen on-disk sanity checks for
  SIT/NAT bitmaps, read-only curseg entries, and device aliasing.

  Additional stability patches address online resize, roll-forward
  recovery, atomic file operations, quota handling, and dirty inode
  syncing livelocks.

  Enhancements:
   - implement dedicated metadata cache for meta, node, and compress
     blocks to decouple metadata from page cache for large folio and
     page != block support
   - dynamically describe SIT, NAT, orphan, dentry, node, and xattr
     block layouts and parameterize block/sector/node conversion macros
   - opt in to multigrain timestamps
   - introduce max_atc_write_bio_entry_cnt sysfs node
   - expose resizable_tail feature via sysfs
   - avoid unnecessary reads in f2fs_update_meta_block()

Bug fixes:
   - compress: force out-place update for all writes, allow pinning
     empty compressed files, and fix truncate/writeback and cache
     population races
   - zone: avoid bio split and GC target block grab failure on
     sequential zones
   - quota: convert inline quota files on quotaon, fix flush failure
     during freeze, and restore empty string mount options
   - atomic/mmap: disallow mmap write and data-modifying fallocate on
     atomic files, avoid truncate_inode_pages_final() on live inodes,
     and fix races in f2fs_vm_page_mkwrite()
   - resize/alias: refresh pinned allocation boundary and flush SIT
     journal during resize, and drop pending discards before reserving
     device alias
   - sanity check: bound SIT/NAT bitmaps by checkpoint pack, check all
     cur_*_segno[] on readonly images, and validate multi-device on
     device alias
   - recovery/cp: release removed xattr node in roll-forward recovery,
     invalidate curseg on change_curseg() failure, and return -EIO for
     lost inode block after cp_error
   - fix livelock in syncing dirty inodes and preserve APPEND/UPDATE ino
     entries
   - use bio_in_atomic() in f2fs_{read,write}_end_io()
   - fix 32-bit shift truncation in mount option bitmasks and defragment
     range

  And a number of minor bug fixes"

* tag 'f2fs-for-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs: (79 commits)
  f2fs: compress: don't overwrite populated compress cache entry
  f2fs: fix to update superblock crc in sanity_check_area_boundary()
  f2fs: fix to fetch ino via ino_of_node() under cache lock
  f2fs: zone: fix to don't fail to grab GC target block on zoned device
  f2fs: don't use truncate_inode_pages_final() on live inodes
  f2fs: expose resizable_tail feature via sysfs
  f2fs: use BIT_ULL for mount option bitmasks
  f2fs: fix submit accounting for IPU writeback
  f2fs: fix to avoid potential reference count leak in do_garbage_collect()
  f2fs: refresh pinned allocation boundary after resize
  f2fs: flush SIT journal during filesystem resize
  f2fs: release a removed xattr node in roll-forward recovery
  f2fs: quota: fix quota flush failure during filesystem freeze
  f2fs: return -EIO for a lost inode block after cp_error
  f2fs: avoid unnecessary reads in f2fs_update_meta_block()
  f2fs: bound the SIT/NAT bitmaps by the checkpoint pack
  f2fs: opt in to multigrain timestamps
  f2fs: fix to use bio_in_atomic() in f2fs_read_end_io()
  f2fs: fix to use bio_in_atomic() in f2fs_write_end_io()
  f2fs: fix to invalidate curseg if change_curseg() fails
  ...
This commit is contained in:
Linus Torvalds 2026-10-27 09:06:00 -10:00
commit 92fcf15176
34 changed files with 4900 additions and 2973 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
Documentation/ABI/testing/sysfs-fs-f2fs
View File

@ -271,7 +271,7 @@ Description: Shows all enabled kernel features.
inode_crtime, lost_found, verity, sb_checksum,
casefold, readonly, compression, test_dummy_encryption_v2,
atomic_write, pin_file, encrypted_casefold, linear_lookup,
fserror.
fserror, metadata_cache, resizable_tail.
What: /sys/fs/f2fs/<disk>/inject_rate
Date: May 2016
@ -1014,9 +1014,48 @@ Description: Every time a write operation completes f2fs_write_end_io() is
(atc) context. The default value for this attribute is UINT_MAX
which means that this functionality is disabled by default.
What: /sys/fs/f2fs/<disk>/max_atc_write_bio_entry_cnt
Date: October 2026
Contact: Chao Yu <chao@kernel.org>
Description: Every time a write operation completes f2fs_write_end_io() is
called. This function may be called from an atomic context,
e.g. from inside an interrupt handler. This attribute controls
the maximum number of entries (folios or cached blocks) in a
write bio that is completed in atomic (atc) context. The default
value for this attribute is UINT_MAX which means that this
functionality is disabled by default.
What: /sys/fs/f2fs/<disk>/pinned_area_max_secno
Date: August 2026
Contact: "Daeho Jeong" <daehojeong@google.com>
Description: This is a read-only entry to show the upper bound section number
for pinned files. Pinned files will only be allocated within
sections 0 to pinned_area_max_secno - 1.
What: /sys/fs/f2fs/<disk>/cache_wb_interval
Date: August 2026
Contact: "Chao Yu" <chao@kernel.org>
Description: This is a writable entry to control writeback interval of
f2fs_writeback-x:y, the range is [100, 30000], by default the value
is 5000, unit is ms.
What: /sys/fs/f2fs/<disk>/cache_wb_dirty_threshold
Date: October 2026
Contact: "Chao Yu" <chao@kernel.org>
Description: This is a writable entry for metadata cache. f2fs_writeback-x:y
will be woken up for writeback once the number of dirty meta and
node caches reaches this value. 0 disables this check. By default
the value is 8192, unit is cache entries.
What: /sys/fs/f2fs/<disk>/cache_wb_total_threshold
Date: October 2026
Contact: "Chao Yu" <chao@kernel.org>
Description: This is a writable entry for metadata cache. f2fs_writeback-x:y
will be woken up for writeback only if the total number of cached
entries also reaches this value. 0 disables this check. By default
the value is 0, unit is cache entries.
If both cache_wb_dirty_threshold and cache_wb_total_threshold are
0, f2fs_writeback-x:y only runs every cache_wb_interval, unless
checkpoint is disabled, in which case it will still be woken up
once there are enough dirty caches to be gathered for writeback.

3
Documentation/filesystems/f2fs.rst
View File

@ -421,7 +421,8 @@ resizable_tail_secno=%u Control the number of sections at the tail of the
(MAIN_SECS - resizable_tail_secno) - 1. If set to 0
(default), there is no tail restriction unless running
on a zoned block device where conventional zones are
used.
used. Online shrink consumes this reservation, reducing
the value by the number of removed sections, down to zero.
======================== ============================================================
Debugfs Entries

2
MAINTAINERS
View File

@ -10169,7 +10169,7 @@ S: Maintained
W: https://f2fs.wiki.kernel.org/
Q: https://patchwork.kernel.org/project/f2fs/list/
B: https://bugzilla.kernel.org/enter_bug.cgi?product=File%20System&component=f2fs
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git dev
F: Documentation/ABI/testing/sysfs-fs-f2fs
F: Documentation/filesystems/f2fs.rst
F: fs/f2fs/

2
fs/f2fs/Makefile
View File

@ -3,7 +3,7 @@ obj-$(CONFIG_F2FS_FS) += f2fs.o
f2fs-y := dir.o file.o inode.o namei.o hash.o super.o inline.o
f2fs-y += checkpoint.o gc.o data.o node.o segment.o recovery.o
f2fs-y += shrinker.o extent_cache.o sysfs.o
f2fs-y += shrinker.o extent_cache.o sysfs.o cache.o
f2fs-$(CONFIG_F2FS_STAT_FS) += debug.o
f2fs-$(CONFIG_F2FS_FS_XATTR) += xattr.o
f2fs-$(CONFIG_F2FS_FS_POSIX_ACL) += acl.o

26
fs/f2fs/acl.c
View File

@ -181,7 +181,7 @@ static void *f2fs_acl_to_disk(struct f2fs_sb_info *sbi,
}
static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type,
struct folio *dfolio)
struct f2fs_cached_block *entry)
{
int name_index = F2FS_XATTR_INDEX_POSIX_ACL_DEFAULT;
void *value = NULL;
@ -191,13 +191,13 @@ static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type,
if (type == ACL_TYPE_ACCESS)
name_index = F2FS_XATTR_INDEX_POSIX_ACL_ACCESS;
retval = f2fs_getxattr(inode, name_index, "", NULL, 0, dfolio);
retval = f2fs_getxattr(inode, name_index, "", NULL, 0, entry);
if (retval > 0) {
value = f2fs_kmalloc(F2FS_I_SB(inode), retval, GFP_F2FS_ZERO);
if (!value)
return ERR_PTR(-ENOMEM);
retval = f2fs_getxattr(inode, name_index, "", value,
retval, dfolio);
retval, entry);
}
if (retval > 0)
@ -242,7 +242,7 @@ static int f2fs_acl_update_mode(const struct mnt_idmap *idmap,
static int __f2fs_set_acl(const struct mnt_idmap *idmap,
struct inode *inode, int type,
struct posix_acl *acl, struct folio *ifolio)
struct posix_acl *acl, struct f2fs_cached_block *ientry)
{
int name_index;
void *value = NULL;
@ -253,7 +253,7 @@ static int __f2fs_set_acl(const struct mnt_idmap *idmap,
switch (type) {
case ACL_TYPE_ACCESS:
name_index = F2FS_XATTR_INDEX_POSIX_ACL_ACCESS;
if (acl && !ifolio) {
if (acl && !ientry) {
error = f2fs_acl_update_mode(idmap, inode, &mode, &acl);
if (error)
return error;
@ -279,7 +279,7 @@ static int __f2fs_set_acl(const struct mnt_idmap *idmap,
}
}
error = f2fs_setxattr(inode, name_index, "", value, size, ifolio, 0);
error = f2fs_setxattr(inode, name_index, "", value, size, ientry, 0);
kfree(value);
if (!error)
@ -374,7 +374,7 @@ static int f2fs_acl_create_masq(struct posix_acl *acl, umode_t *mode_p)
static int f2fs_acl_create(struct inode *dir, umode_t *mode,
struct posix_acl **default_acl, struct posix_acl **acl,
struct folio *dfolio)
struct f2fs_cached_block *entry)
{
struct posix_acl *p;
struct posix_acl *clone;
@ -386,7 +386,7 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode,
if (S_ISLNK(*mode) || !IS_POSIXACL(dir))
return 0;
p = __f2fs_get_acl(dir, ACL_TYPE_DEFAULT, dfolio);
p = __f2fs_get_acl(dir, ACL_TYPE_DEFAULT, entry);
if (!p || p == ERR_PTR(-EOPNOTSUPP)) {
*mode &= ~current_umask();
return 0;
@ -423,13 +423,13 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode,
return ret;
}
int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio,
struct folio *dfolio)
int f2fs_init_acl(struct inode *inode, struct inode *dir, struct f2fs_cached_block *ientry,
struct f2fs_cached_block *dentry)
{
struct posix_acl *default_acl = NULL, *acl = NULL;
int error;
error = f2fs_acl_create(dir, &inode->i_mode, &default_acl, &acl, dfolio);
error = f2fs_acl_create(dir, &inode->i_mode, &default_acl, &acl, dentry);
if (error)
return error;
@ -437,7 +437,7 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio,
if (default_acl) {
error = __f2fs_set_acl(NULL, inode, ACL_TYPE_DEFAULT,
default_acl, ifolio);
default_acl, ientry);
posix_acl_release(default_acl);
} else {
inode->i_default_acl = NULL;
@ -445,7 +445,7 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct folio *ifolio,
if (acl) {
if (!error)
error = __f2fs_set_acl(NULL, inode, ACL_TYPE_ACCESS,
acl, ifolio);
acl, ientry);
posix_acl_release(acl);
} else {
inode->i_acl = NULL;

8
fs/f2fs/acl.h
View File

@ -36,14 +36,16 @@ struct f2fs_acl_header {
struct posix_acl *f2fs_get_acl(struct inode *, int, bool);
int f2fs_set_acl(const struct mnt_idmap *, struct dentry *,
struct posix_acl *, int);
int f2fs_init_acl(struct inode *, struct inode *, struct folio *ifolio,
struct folio *dfolio);
int f2fs_init_acl(struct inode *inode, struct inode *dir,
struct f2fs_cached_block *ientry,
struct f2fs_cached_block *dentry);
#else
#define f2fs_get_acl NULL
#define f2fs_set_acl NULL
static inline int f2fs_init_acl(struct inode *inode, struct inode *dir,
struct folio *ifolio, struct folio *dfolio)
struct f2fs_cached_block *ientry,
struct f2fs_cached_block *dentry)
{
return 0;
}

894
fs/f2fs/cache.c Normal file
View File

@ -0,0 +1,894 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (c) 2026 Google LLC
* Author: Chao Yu <chaseyu@google.com>
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
#include <linux/radix-tree.h>
#include <linux/slab.h>
#include <linux/list.h>
#include <linux/pagemap.h>
#include <linux/kthread.h>
#include <linux/freezer.h>
#include <linux/delay.h>
#include "f2fs.h"
#include "cache.h"
#include "node.h"
#include <trace/events/f2fs.h>
#include "segment.h"
static void f2fs_free_cache(struct f2fs_cached_block *entry);
void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry,
enum page_type type)
{
/* in case the entry was truncated or on-going shrink */
if (!entry->cache)
return;
if (!f2fs_cache_test_writeback(entry))
return;
/* submit cached bio */
f2fs_submit_merged_write_cache(entry->cache->sbi, entry, 0, type);
wait_on_bit_io(&entry->state, F2FS_BLOCK_WRITEBACK,
TASK_UNINTERRUPTIBLE);
}
void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry)
{
/* in case the entry was truncated or on-going shrink */
if (!entry->cache)
return;
f2fs_cache_wait_writeback_cond(entry,
IS_META_CACHE(entry->cache) ? META : NODE);
}
void f2fs_cache_update_tag(struct f2fs_cached_block *entry,
unsigned int clear_from, unsigned int set_to)
{
struct f2fs_cached_block_list *cache = entry->cache;
unsigned long flags;
spin_lock_irqsave(&cache->tree_lock, flags);
if (clear_from != F2FS_CACHE_TAG_NONE)
radix_tree_tag_clear(&cache->root, entry->index, clear_from);
if (set_to != F2FS_CACHE_TAG_NONE)
radix_tree_tag_set(&cache->root, entry->index, set_to);
spin_unlock_irqrestore(&cache->tree_lock, flags);
}
bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry)
{
struct f2fs_cached_block_list *cache = entry->cache;
f2fs_cache_set_uptodate(entry);
#ifdef CONFIG_F2FS_CHECK_FS
if (f2fs_is_node_cache(entry) && IS_INODE(cache->sbi, entry))
f2fs_inode_chksum_set(cache->sbi, entry);
#endif
if (!f2fs_cache_test_and_set_dirty(entry)) {
enum count_type type = IS_META_CACHE(cache) ?
F2FS_DIRTY_META : F2FS_DIRTY_NODES;
trace_f2fs_cache_set_dirty(entry,
IS_META_CACHE(cache) ? META : NODE);
f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_NONE,
F2FS_CACHE_TAG_DIRTY);
inc_cache_count(cache->sbi, type);
f2fs_wake_up_cache_wb(cache->sbi);
return true;
}
return false;
}
void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry)
{
struct f2fs_cached_block_list *cache = entry->cache;
enum count_type type = IS_META_CACHE(cache) ?
F2FS_DIRTY_META : F2FS_DIRTY_NODES;
f2fs_cache_clear_uptodate(entry);
if (!f2fs_cache_test_and_clear_dirty(entry))
return;
f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY,
F2FS_CACHE_TAG_NONE);
dec_cache_count(cache->sbi, type);
}
void f2fs_start_cache_writeback(struct f2fs_cached_block *entry)
{
f2fs_cache_set_writeback(entry);
f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_DIRTY,
F2FS_CACHE_TAG_WRITEBACK);
}
/*
* The temporary references taken around clear_and_wake_up_bit() are counted
* in units of F2FS_CACHE_PIN_BIAS, like GUP_PIN_COUNTING_BIAS for folios, so
* that they can be told apart from the regular references.
*/
#define F2FS_CACHE_PIN_BIAS (1 << 16)
static void f2fs_cache_pin(struct f2fs_cached_block *entry)
{
atomic_add(F2FS_CACHE_PIN_BIAS, &entry->refcount);
}
static void f2fs_cache_unpin(struct f2fs_cached_block *entry)
{
int ref = atomic_sub_return(F2FS_CACHE_PIN_BIAS, &entry->refcount);
WARN_ON_ONCE(ref < 0);
if (!ref)
f2fs_free_cache(entry);
}
void f2fs_end_cache_writeback(struct f2fs_cached_block *entry)
{
/*
* should call f2fs_cache_update_tag() before clearing writeback bit,
* in case f2fs_truncate_cache() set entry->cache to NULL.
*/
f2fs_cache_update_tag(entry, F2FS_CACHE_TAG_WRITEBACK,
F2FS_CACHE_TAG_NONE);
/*
* Writeback does not hold an entry reference of its own, relying
* on truncation to wait for the clearing of F2FS_BLOCK_WRITEBACK.
* But here we must make sure that the entry is not freed and
* reused before clear_and_wake_up_bit().
*/
f2fs_cache_pin(entry);
clear_and_wake_up_bit(F2FS_BLOCK_WRITEBACK, &entry->state);
f2fs_cache_unpin(entry);
}
static int f2fs_cache_refcount(struct f2fs_cached_block *entry)
{
return atomic_read(&entry->refcount);
}
/* the number of regular references, excluding the temporary ones */
static int f2fs_cache_users(struct f2fs_cached_block *entry)
{
return f2fs_cache_refcount(entry) & (F2FS_CACHE_PIN_BIAS - 1);
}
static void f2fs_do_free_cache(struct f2fs_cached_block *entry)
{
kfree(entry->data);
kfree(entry);
}
static void f2fs_free_cache(struct f2fs_cached_block *entry)
{
WARN_ON_ONCE(!list_empty(&entry->list));
WARN_ON_ONCE(f2fs_cache_refcount(entry));
f2fs_do_free_cache(entry);
}
void f2fs_cache_get(struct f2fs_cached_block *entry)
{
atomic_inc(&entry->refcount);
}
static bool f2fs_cache_put(struct f2fs_cached_block *entry)
{
WARN_ON_ONCE(!f2fs_cache_refcount(entry));
if (atomic_dec_and_test(&entry->refcount)) {
f2fs_free_cache(entry);
return true;
}
return false;
}
static struct f2fs_cached_block *f2fs_create_cache(
struct f2fs_cached_block_list *cache,
unsigned long index, bool nofail)
{
struct f2fs_sb_info *sbi = cache->sbi;
struct f2fs_cached_block *entry;
unsigned int flags = GFP_NOFS;
if (index == ULONG_MAX)
return ERR_PTR(-ERANGE);
if (nofail) {
flags |= __GFP_NOFAIL;
entry = kzalloc_obj(*entry, flags);
entry->data = kmalloc(sbi->blocksize, flags);
} else {
entry = f2fs_kzalloc(sbi, sizeof(*entry), flags);
if (!entry)
return ERR_PTR(-ENOMEM);
entry->data = f2fs_kmalloc(sbi, sbi->blocksize, flags);
if (!entry->data) {
kfree(entry);
return ERR_PTR(-ENOMEM);
}
}
entry->index = index;
atomic_set(&entry->refcount, 0);
if (!IS_COMPRESS_CACHE(cache))
entry->next_entry = NULL;
else
entry->ino = 0;
INIT_LIST_HEAD(&entry->list);
entry->cache = cache;
return entry;
}
static struct f2fs_cached_block *f2fs_insert_cache(
struct f2fs_cached_block_list *cache,
unsigned long index,
struct f2fs_cached_block *new)
{
struct f2fs_cached_block *e;
int ret;
unsigned long flags;
ret = radix_tree_preload(GFP_NOFS | __GFP_NOFAIL);
f2fs_bug_on(cache->sbi, ret);
spin_lock(&cache->list_lock);
spin_lock_irqsave(&cache->tree_lock, flags);
e = radix_tree_lookup(&cache->root, index);
if (!e) {
e = new;
f2fs_bug_on(cache->sbi, f2fs_cache_refcount(e));
ret = radix_tree_insert(&cache->root, index, e);
f2fs_bug_on(cache->sbi, ret);
/* radix tree referenced cache entry */
f2fs_cache_get(e);
f2fs_bug_on(cache->sbi, !list_empty(&e->list));
list_add_tail(&e->list, &cache->lru_list);
cache->num_entries++;
}
f2fs_cache_get(e);
spin_unlock_irqrestore(&cache->tree_lock, flags);
spin_unlock(&cache->list_lock);
radix_tree_preload_end();
if (new != e) {
f2fs_bug_on(cache->sbi, f2fs_cache_refcount(new));
f2fs_do_free_cache(new);
}
return e;
}
struct f2fs_cached_block *f2fs_find_cache(
struct f2fs_cached_block_list *cache,
unsigned long index,
enum f2fs_cache_request_flag rflag)
{
struct f2fs_cached_block *entry;
unsigned long flags;
bool access = rflag & F2FS_CACHE_ACCESS;
spin_lock_irqsave(&cache->tree_lock, flags);
entry = radix_tree_lookup(&cache->root, index);
if (entry) {
f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry));
f2fs_cache_get(entry);
if (access && !f2fs_cache_test_referenced(entry))
f2fs_cache_set_referenced(entry);
} else {
entry = ERR_PTR(-ENOENT);
}
spin_unlock_irqrestore(&cache->tree_lock, flags);
return entry;
}
struct f2fs_cached_block *f2fs_grab_cache(
struct f2fs_cached_block_list *cache,
unsigned long index, int flags)
{
struct f2fs_cached_block *entry, *new;
bool create = flags & F2FS_CACHE_CREATE;
bool nofail = flags & F2FS_CACHE_NOFAIL;
bool lock = flags & F2FS_CACHE_LOCK;
repeat:
entry = f2fs_find_cache(cache, index, F2FS_CACHE_ACCESS);
if (!IS_ERR(entry))
goto found;
if (!create)
return ERR_PTR(-ENOENT);
new = f2fs_create_cache(cache, index, nofail);
if (IS_ERR(new))
return new;
entry = f2fs_insert_cache(cache, index, new);
found:
if (lock) {
f2fs_lock_cache(entry);
/* has been truncated */
if (entry->cache != cache) {
f2fs_put_cache(entry, true);
goto repeat;
}
}
return entry;
}
bool f2fs_trylock_cache(struct f2fs_cached_block *entry)
{
return !test_and_set_bit(F2FS_BLOCK_LOCKED, &entry->state);
}
void f2fs_lock_cache(struct f2fs_cached_block *entry)
{
wait_on_bit_lock(&entry->state, F2FS_BLOCK_LOCKED,
TASK_UNINTERRUPTIBLE);
}
void f2fs_unlock_cache(struct f2fs_cached_block *entry)
{
/*
* In asynchronous read I/O completion (e.g. from f2fs_ra_node_cache()),
* the I/O completion does not hold a reference of its own. Once
* clear_and_wake_up_bit() clears F2FS_BLOCK_LOCKED, a concurrent waiter
* in f2fs_lock_cache() (e.g. from f2fs_truncate_cache()) can wake up,
* truncate the entry, and drop the final reference before wake_up_bit()
* finishes.
* Pin the entry here to make sure it is not freed before wake_up_bit()
* completes.
*/
f2fs_cache_pin(entry);
clear_and_wake_up_bit(F2FS_BLOCK_LOCKED, &entry->state);
f2fs_cache_unpin(entry);
}
bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock)
{
if (IS_ERR_OR_NULL(entry))
return false;
if (unlock)
f2fs_unlock_cache(entry);
return f2fs_cache_put(entry);
}
unsigned int f2fs_cache_gang_lookup(struct f2fs_cached_block_list *cache,
struct f2fs_cached_block **entries,
pgoff_t *index, unsigned long end)
{
unsigned long flags;
unsigned int max_nr = min((unsigned long)F2FS_ONSTACK_CACHES, end - *index);
int nr, i;
if (*index >= end || *index == ULONG_MAX)
return 0;
spin_lock_irqsave(&cache->tree_lock, flags);
nr = radix_tree_gang_lookup(&cache->root, (void **)entries,
*index, max_nr);
if (!nr)
goto out_unlock;
for (i = 0; i < nr; i++) {
struct f2fs_cached_block *entry = entries[i];
if (entry->index >= end) {
nr = i;
break;
}
f2fs_cache_get(entry);
}
if (nr)
*index = entries[nr - 1]->index + 1;
out_unlock:
spin_unlock_irqrestore(&cache->tree_lock, flags);
return nr;
}
unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache,
struct f2fs_cached_block **entries,
pgoff_t *index, unsigned int max_nr,
int tag)
{
unsigned long flags;
int nr, i;
if (*index == ULONG_MAX)
return 0;
spin_lock_irqsave(&cache->tree_lock, flags);
nr = radix_tree_gang_lookup_tag(&cache->root, (void **)entries,
*index, max_nr, tag);
if (!nr)
goto out;
for (i = 0; i < nr; i++)
f2fs_cache_get(entries[i]);
*index = entries[nr - 1]->index + 1;
out:
spin_unlock_irqrestore(&cache->tree_lock, flags);
return nr;
}
void f2fs_cache_gang_release(struct f2fs_cached_block **entries,
unsigned int nr_entries)
{
int i;
for (i = 0; i < nr_entries; i++)
f2fs_put_cache(entries[i], false);
}
void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache)
{
unsigned long index = 0;
struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES];
int nr, i;
next:
nr = f2fs_cache_gang_lookup_tag(cache, entries, &index,
F2FS_ONSTACK_CACHES, F2FS_CACHE_TAG_WRITEBACK);
if (!nr)
return;
for (i = 0; i < nr; i++)
f2fs_cache_wait_writeback(entries[i]);
f2fs_cache_gang_release(entries, nr);
cond_resched();
goto next;
}
static void f2fs_do_truncate_cache(struct f2fs_cached_block *entry,
bool drop_dirty)
{
struct f2fs_cached_block_list *cache = entry->cache;
unsigned long flags;
if (drop_dirty)
goto drop_it;
if (f2fs_cache_test_dirty(entry) ||
f2fs_cache_test_writeback(entry))
return;
drop_it:
f2fs_cache_wait_writeback(entry);
f2fs_drop_cache_dirty(entry);
spin_lock(&cache->list_lock);
spin_lock_irqsave(&cache->tree_lock, flags);
f2fs_bug_on(cache->sbi, !entry->cache);
if (!radix_tree_delete(&cache->root, entry->index))
f2fs_bug_on(cache->sbi, !entry->cache);
entry->cache = NULL;
cache->num_entries--;
atomic_dec(&entry->refcount);
f2fs_bug_on(cache->sbi, !f2fs_cache_refcount(entry));
f2fs_bug_on(cache->sbi, list_empty(&entry->list));
list_del_init(&entry->list);
spin_unlock_irqrestore(&cache->tree_lock, flags);
spin_unlock(&cache->list_lock);
}
void f2fs_truncate_locked_cache(struct f2fs_cached_block *entry,
bool drop_dirty)
{
if (!entry->cache)
return;
f2fs_do_truncate_cache(entry, drop_dirty);
}
void f2fs_truncate_cache(struct f2fs_cached_block *entry, bool drop_dirty)
{
f2fs_lock_cache(entry);
f2fs_truncate_locked_cache(entry, drop_dirty);
f2fs_unlock_cache(entry);
}
static void f2fs_drop_cache(struct f2fs_cached_block_list *cache,
block_t blkaddr, bool drop_dirty)
{
struct f2fs_cached_block *entry;
entry = f2fs_find_cache(cache, blkaddr, 0);
if (IS_ERR(entry))
return;
f2fs_truncate_cache(entry, drop_dirty);
f2fs_put_cache(entry, false);
}
void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache,
unsigned long start, unsigned long len, bool drop_dirty)
{
unsigned long index = start;
unsigned long end = (ULONG_MAX - start < len) ?
ULONG_MAX : (start + len);
struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES];
int nr, i;
if (len == 1)
return f2fs_drop_cache(cache, index, drop_dirty);
next:
nr = f2fs_cache_gang_lookup(cache, entries, &index, end);
if (!nr)
return;
for (i = 0; i < nr; i++)
f2fs_truncate_cache(entries[i], drop_dirty);
f2fs_cache_gang_release(entries, nr);
if (index < end) {
cond_resched();
goto next;
}
}
int f2fs_init_cache(struct f2fs_sb_info *sbi,
struct f2fs_cached_block_list *cache,
enum f2fs_cache_type type)
{
cache->sbi = sbi;
cache->type = type;
INIT_RADIX_TREE(&cache->root, GFP_ATOMIC);
spin_lock_init(&cache->tree_lock);
spin_lock_init(&cache->list_lock);
INIT_LIST_HEAD(&cache->lru_list);
cache->num_entries = 0;
return 0;
}
void f2fs_destroy_cache(struct f2fs_cached_block_list *cache)
{
struct list_head *head = &cache->lru_list;
struct f2fs_cached_block *entry;
unsigned long flags;
f2fs_cache_wait_on_all_writeback(cache);
next:
spin_lock(&cache->list_lock);
if (list_empty(head)) {
spin_unlock(&cache->list_lock);
return;
}
entry = list_first_entry(head, struct f2fs_cached_block, list);
spin_lock_irqsave(&cache->tree_lock, flags);
radix_tree_delete(&cache->root, entry->index);
cache->num_entries--;
list_del_init(&entry->list);
spin_unlock_irqrestore(&cache->tree_lock, flags);
spin_unlock(&cache->list_lock);
/* wait on read cache IO */
f2fs_lock_cache(entry);
/* wait on write cache IO */
f2fs_cache_wait_writeback(entry);
f2fs_bug_on(cache->sbi, f2fs_cache_test_dirty(entry));
f2fs_bug_on(cache->sbi, f2fs_cache_test_writeback(entry));
f2fs_bug_on(cache->sbi, !list_empty(&entry->list));
/* the I/O completion may not have unpinned the entry yet */
f2fs_bug_on(cache->sbi, f2fs_cache_users(entry) != 1);
f2fs_put_cache(entry, true);
goto next;
}
static unsigned long f2fs_do_shrink_cache(struct f2fs_cached_block_list *cache,
unsigned long nr_to_scan)
{
struct f2fs_cached_block *entry, *next;
LIST_HEAD(dispose_list);
LIST_HEAD(keep_list);
unsigned long freed = 0;
unsigned long scanned = 0;
/* Phase 1: Isolate candidate entries from LRU list into dispose_list */
spin_lock(&cache->list_lock);
list_for_each_entry_safe(entry, next, &cache->lru_list, list) {
if (scanned >= cache->num_entries)
break;
if (scanned++ >= nr_to_scan)
break;
/* If accessed, give it a second chance to rotate to tail */
if (f2fs_cache_test_and_clear_referenced(entry)) {
list_move_tail(&entry->list, &cache->lru_list);
continue;
}
if (f2fs_cache_test_dirty(entry) ||
f2fs_cache_test_writeback(entry) ||
f2fs_cache_test_locked(entry))
continue;
if (f2fs_cache_refcount(entry) != 1)
continue;
list_move_tail(&entry->list, &dispose_list);
}
spin_unlock(&cache->list_lock);
/* Phase 2: Process isolated candidates one by one */
while (1) {
spin_lock(&cache->list_lock);
entry = list_first_entry_or_null(&dispose_list,
struct f2fs_cached_block, list);
if (!entry) {
spin_unlock(&cache->list_lock);
break;
}
f2fs_cache_get(entry);
list_move_tail(&entry->list, &keep_list);
spin_unlock(&cache->list_lock);
if (!f2fs_trylock_cache(entry)) {
f2fs_put_cache(entry, false);
continue;
}
/* the entry has been truncated */
if (!entry->cache) {
f2fs_put_cache(entry, true);
continue;
}
/*
* at least there are shrinker, radix tree and another user
* has referenced the entry.
*/
if (f2fs_cache_refcount(entry) >= 3) {
f2fs_put_cache(entry, true);
continue;
}
f2fs_do_truncate_cache(entry, false);
if (f2fs_put_cache(entry, true))
freed++;
}
/* Phase 3: Splice un-reclaimed entries back onto cache->lru_list */
if (!list_empty(&keep_list)) {
spin_lock(&cache->list_lock);
list_splice_tail(&keep_list, &cache->lru_list);
spin_unlock(&cache->list_lock);
}
return freed;
}
unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi,
unsigned long nr_to_scan)
{
unsigned long freed;
freed = f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan);
if (freed >= nr_to_scan)
return freed;
freed += f2fs_do_shrink_cache(NODE_CACHE(sbi), nr_to_scan - freed);
if (freed >= nr_to_scan)
return freed;
freed += f2fs_do_shrink_cache(COMPRESS_CACHE(sbi), nr_to_scan - freed);
return freed;
}
unsigned long f2fs_shrink_meta_cache(struct f2fs_sb_info *sbi,
unsigned long nr_to_scan)
{
return f2fs_do_shrink_cache(META_CACHE(sbi), nr_to_scan);
}
unsigned long f2fs_shrink_node_cache(struct f2fs_sb_info *sbi,
unsigned long nr_to_scan)
{
return f2fs_do_shrink_cache(NODE_CACHE(sbi), nr_to_scan);
}
static inline unsigned long f2fs_total_cached_entries(struct f2fs_sb_info *sbi)
{
unsigned long total = READ_ONCE(META_CACHE(sbi)->num_entries) +
READ_ONCE(NODE_CACHE(sbi)->num_entries);
#ifdef CONFIG_F2FS_FS_COMPRESSION
if (test_opt(sbi, COMPRESS_CACHE))
total += READ_ONCE(COMPRESS_CACHE(sbi)->num_entries);
#endif
return total;
}
static inline s64 f2fs_nr_dirty_caches(struct f2fs_sb_info *sbi)
{
return get_nr_caches(sbi, F2FS_DIRTY_META) +
get_nr_caches(sbi, F2FS_DIRTY_NODES);
}
/*
* f2fs_write_{meta,node}_caches() skip writeback until there are enough
* dirty caches to gather, so don't wake up f2fs_writeback for nothing.
*/
static inline bool f2fs_cache_wb_collectable(struct f2fs_sb_info *sbi,
s64 nr_dirty, int type)
{
return nr_dirty && nr_dirty >= nr_caches_to_skip(sbi, type);
}
static bool f2fs_cache_wb_exceeded(struct f2fs_sb_info *sbi)
{
struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread;
unsigned int total_threshold, dirty_threshold;
s64 nr_meta, nr_node;
total_threshold = READ_ONCE(cache_thread->cache_wb_total_threshold);
dirty_threshold = READ_ONCE(cache_thread->cache_wb_dirty_threshold);
/*
* Both thresholds are disabled, rely on periodic writeback only,
* unless checkpoint is disabled, as nothing else will flush dirty
* caches then.
*/
if (!total_threshold && !dirty_threshold &&
!is_sbi_flag_set(sbi, SBI_CP_DISABLED))
return false;
if (total_threshold &&
f2fs_total_cached_entries(sbi) < total_threshold)
return false;
nr_meta = get_nr_caches(sbi, F2FS_DIRTY_META);
nr_node = get_nr_caches(sbi, F2FS_DIRTY_NODES);
if (dirty_threshold && nr_meta + nr_node < dirty_threshold)
return false;
return f2fs_cache_wb_collectable(sbi, nr_meta, META) ||
f2fs_cache_wb_collectable(sbi, nr_node, NODE);
}
static bool f2fs_should_wake_up_cache_wb(struct f2fs_sb_info *sbi)
{
/* last round made no progress, wait for periodic writeback */
if (READ_ONCE(sbi->cache_thread.cache_wb_throttled))
return false;
return f2fs_cache_wb_exceeded(sbi);
}
static int f2fs_cache_writeback_kthread(void *data)
{
struct f2fs_sb_info *sbi = data;
struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread;
wait_queue_head_t *wq = &cache_thread->cache_wb_wq;
set_freezable();
while (!kthread_should_stop()) {
unsigned int interval = READ_ONCE(cache_thread->cache_wb_interval);
long nr_meta = 0, nr_node = 0;
s64 nr_dirty;
wait_event_freezable_timeout(*wq,
kthread_should_stop() ||
f2fs_should_wake_up_cache_wb(sbi),
msecs_to_jiffies(interval));
if (kthread_should_stop())
break;
/*
* If thresholds are exceeded, throttle threshold based wakeup
* until a round makes progress, otherwise we may busy-loop if
* this round can not write anything, e.g. on cp_error, freezing,
* frozen fs, or lock contention.
*/
WRITE_ONCE(cache_thread->cache_wb_throttled,
f2fs_cache_wb_exceeded(sbi));
if (f2fs_readonly(sbi->sb))
continue;
if (f2fs_cp_error(sbi))
continue;
if (unlikely(freezing(current)))
continue;
if (!sb_start_write_trylock(sbi->sb))
continue;
nr_dirty = f2fs_nr_dirty_caches(sbi);
f2fs_write_meta_caches(sbi, &nr_meta);
f2fs_write_node_caches(sbi, &nr_node);
sb_end_write(sbi->sb);
/*
* Don't rely on the delta of dirty caches only, as caches may
* be dirtied concurrently; also count in caches cleaned by
* others, e.g. checkpoint triggered by f2fs_balance_fs_bg().
*/
if (nr_meta || nr_node || f2fs_nr_dirty_caches(sbi) < nr_dirty)
WRITE_ONCE(cache_thread->cache_wb_throttled, false);
cond_resched();
}
return 0;
}
void f2fs_wake_up_cache_wb(struct f2fs_sb_info *sbi)
{
struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread;
/* pairs with smp_store_release() in f2fs_start_cache_wb_thread() */
if (!smp_load_acquire(&cache_thread->cache_wb_task))
return;
if (!f2fs_should_wake_up_cache_wb(sbi))
return;
if (wq_has_sleeper(&cache_thread->cache_wb_wq))
wake_up(&cache_thread->cache_wb_wq);
}
int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi)
{
struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread;
struct task_struct *task;
dev_t dev = sbi->sb->s_dev;
char name[36];
if (cache_thread->cache_wb_task)
return 0;
init_waitqueue_head(&cache_thread->cache_wb_wq);
cache_thread->cache_wb_interval = DEF_DIRTY_CACHE_TIMEOUT;
cache_thread->cache_wb_dirty_threshold = DEF_CACHE_WB_DIRTY_THRESH;
cache_thread->cache_wb_total_threshold = 0;
cache_thread->cache_wb_throttled = false;
snprintf(name, sizeof(name), "f2fs_writeback-%u:%u",
MAJOR(dev), MINOR(dev));
task = kthread_run(f2fs_cache_writeback_kthread, sbi, "%s", name);
if (IS_ERR(task))
return PTR_ERR(task);
/* pairs with smp_load_acquire() in f2fs_wake_up_cache_wb() */
smp_store_release(&cache_thread->cache_wb_task, task);
return 0;
}
void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi)
{
struct f2fs_cache_kthread *cache_thread = &sbi->cache_thread;
if (!cache_thread->cache_wb_task)
return;
kthread_stop(cache_thread->cache_wb_task);
WRITE_ONCE(cache_thread->cache_wb_task, NULL);
}

252
fs/f2fs/cache.h Normal file
View File

@ -0,0 +1,252 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (c) 2026 Google LLC
* Author: Chao Yu <chaseyu@google.com>
*/
#ifndef _LINUX_F2FS_CACHE_H
#define _LINUX_F2FS_CACHE_H
#include <linux/pagemap.h>
#include <linux/mm.h>
#include <linux/list.h>
#include <linux/radix-tree.h>
#include <linux/spinlock.h>
#include <linux/wait.h>
#include <linux/types.h>
struct f2fs_rwsem;
struct f2fs_io_info;
enum page_type;
/* Represents a single cached block (meta, node or compress) */
struct f2fs_cached_block {
struct list_head list; /* LRU list head */
struct f2fs_cached_block_list *cache; /* parent cache list */
union {
struct f2fs_cached_block *next_entry;/* chain for merged BIO */
nid_t ino; /* inode number for compress cache */
};
unsigned long index; /* key in radix tree, (meta/compress: pba, node: nid) */
unsigned long state; /* cache entry state (e.g., Dirty, UpToDate) */
void *data; /* blocksize-aligned memory (4KB or 16KB) */
atomic_t refcount; /* reference count */
};
struct f2fs_sb_info;
enum f2fs_cache_type {
F2FS_META_CACHE,
F2FS_NODE_CACHE,
F2FS_COMPRESS_CACHE,
NR_CACHE_TYPES,
};
/* Main cache control structure (per sb_info) */
struct f2fs_cached_block_list {
struct f2fs_sb_info *sbi; /* Pointer to f2fs_sb_info */
struct radix_tree_root root; /* Radix tree for cache lookup */
spinlock_t tree_lock; /* Lock for radix tree */
struct list_head lru_list; /* Single global LRU list */
spinlock_t list_lock; /* Lock for LRU list */
enum f2fs_cache_type type; /* Cache type (Node, Meta, Compress) */
unsigned long num_entries; /* Current number of entries */
};
#define IS_META_CACHE(cache) ((cache)->type == F2FS_META_CACHE)
#define IS_NODE_CACHE(cache) ((cache)->type == F2FS_NODE_CACHE)
#define IS_COMPRESS_CACHE(cache) ((cache)->type == F2FS_COMPRESS_CACHE)
/* Flags for f2fs_cached_block state */
enum f2fs_cached_state {
F2FS_BLOCK_LOCKED, /* cache entry is locked */
F2FS_BLOCK_UPTODATE, /* cache data is valid */
F2FS_BLOCK_DIRTY, /* cache data is dirty, need to writeback the data */
F2FS_BLOCK_WRITEBACK, /* cache data is writeback state */
F2FS_BLOCK_REFERENCED, /* cache was accessed recently, shrinker will skip it for once */
F2FS_BLOCK_INLINE_DATA, /* indicate inline data */
};
enum {
__F2FS_CACHE_CREATE, /* create the cache if there is no cache entry */
__F2FS_CACHE_LOCK, /* get and lock the cache entry */
__F2FS_CACHE_NOFAIL, /* do not allow failure */
__F2FS_CACHE_ACCESS, /* give a chance to add referenced tag */
};
enum f2fs_cache_request_flag {
F2FS_CACHE_CREATE = 1 << __F2FS_CACHE_CREATE,
F2FS_CACHE_LOCK = 1 << __F2FS_CACHE_LOCK,
F2FS_CACHE_NOFAIL = 1 << __F2FS_CACHE_NOFAIL,
F2FS_CACHE_ACCESS = 1 << __F2FS_CACHE_ACCESS,
};
#define F2FS_CACHE_LOCK_CREATE (F2FS_CACHE_LOCK | F2FS_CACHE_CREATE)
#define F2FS_ONSTACK_CACHES (32)
#define METADATA_CACHE_SHRINK_NUMBER (128)
#define F2FS_CACHE_FLAG_TEST_FUNC(name, flagname) \
static inline bool f2fs_cache_test_##name( \
const struct f2fs_cached_block *entry) \
{ \
return test_bit(F2FS_BLOCK_##flagname, &entry->state); \
} \
#define F2FS_CACHE_FLAG_SET_FUNC(name, flagname) \
static inline void f2fs_cache_set_##name( \
struct f2fs_cached_block *entry) \
{ \
set_bit(F2FS_BLOCK_##flagname, &entry->state); \
} \
#define F2FS_CACHE_FLAG_CLEAR_FUNC(name, flagname) \
static inline void f2fs_cache_clear_##name( \
struct f2fs_cached_block *entry) \
{ \
clear_bit(F2FS_BLOCK_##flagname, &entry->state); \
} \
#define F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(name, flagname) \
static inline bool f2fs_cache_test_and_set_##name( \
struct f2fs_cached_block *entry) \
{ \
return test_and_set_bit(F2FS_BLOCK_##flagname, &entry->state); \
} \
#define F2FS_CACHE_FLAG_TEST_AND_CLEAR_FUNC(name, flagname) \
static inline bool f2fs_cache_test_and_clear_##name( \
struct f2fs_cached_block *entry) \
{ \
return test_and_clear_bit(F2FS_BLOCK_##flagname, &entry->state);\
} \
F2FS_CACHE_FLAG_TEST_FUNC(locked, LOCKED);
F2FS_CACHE_FLAG_SET_FUNC(locked, LOCKED);
F2FS_CACHE_FLAG_CLEAR_FUNC(locked, LOCKED);
F2FS_CACHE_FLAG_TEST_FUNC(uptodate, UPTODATE);
F2FS_CACHE_FLAG_SET_FUNC(uptodate, UPTODATE);
F2FS_CACHE_FLAG_CLEAR_FUNC(uptodate, UPTODATE);
F2FS_CACHE_FLAG_TEST_FUNC(dirty, DIRTY);
F2FS_CACHE_FLAG_SET_FUNC(dirty, DIRTY);
F2FS_CACHE_FLAG_CLEAR_FUNC(dirty, DIRTY);
F2FS_CACHE_FLAG_TEST_AND_SET_FUNC(dirty, DIRTY);
F2FS_CACHE_FLAG_TEST_AND_CLEAR_FUNC(dirty, DIRTY);
F2FS_CACHE_FLAG_TEST_FUNC(writeback, WRITEBACK);
F2FS_CACHE_FLAG_SET_FUNC(writeback, WRITEBACK);
F2FS_CACHE_FLAG_CLEAR_FUNC(writeback, WRITEBACK);
F2FS_CACHE_FLAG_TEST_FUNC(inline, INLINE_DATA);
F2FS_CACHE_FLAG_SET_FUNC(inline, INLINE_DATA);
F2FS_CACHE_FLAG_CLEAR_FUNC(inline, INLINE_DATA);
F2FS_CACHE_FLAG_TEST_FUNC(referenced, REFERENCED);
F2FS_CACHE_FLAG_SET_FUNC(referenced, REFERENCED);
F2FS_CACHE_FLAG_TEST_AND_CLEAR_FUNC(referenced, REFERENCED);
static inline void *cache_address(const struct f2fs_cached_block *entry)
{
return entry->data;
}
#define CACHED_NODE(entry) ((struct f2fs_node *)(cache_address(entry)))
static inline struct folio *cache_folio(const struct f2fs_cached_block *entry)
{
return virt_to_folio(entry->data);
}
int f2fs_init_cache(struct f2fs_sb_info *sbi,
struct f2fs_cached_block_list *cache,
enum f2fs_cache_type type);
void f2fs_destroy_cache(struct f2fs_cached_block_list *cache);
void f2fs_cache_get(struct f2fs_cached_block *entry);
struct f2fs_cached_block *f2fs_find_cache(
struct f2fs_cached_block_list *cache,
unsigned long index,
enum f2fs_cache_request_flag rflag);
#define F2FS_CACHE_TAG_NONE 0
#define F2FS_CACHE_TAG_DIRTY 1
#define F2FS_CACHE_TAG_WRITEBACK 2
bool f2fs_trylock_cache(struct f2fs_cached_block *entry);
void f2fs_lock_cache(struct f2fs_cached_block *entry);
void f2fs_unlock_cache(struct f2fs_cached_block *entry);
bool f2fs_put_cache(struct f2fs_cached_block *entry, bool unlock);
bool f2fs_mark_cache_dirty(struct f2fs_cached_block *entry);
void f2fs_drop_cache_dirty(struct f2fs_cached_block *entry);
void f2fs_start_cache_writeback(struct f2fs_cached_block *entry);
void f2fs_end_cache_writeback(struct f2fs_cached_block *entry);
unsigned int f2fs_cache_gang_lookup(struct f2fs_cached_block_list *cache,
struct f2fs_cached_block **entries,
pgoff_t *index, unsigned long end);
unsigned int f2fs_cache_gang_lookup_tag(struct f2fs_cached_block_list *cache,
struct f2fs_cached_block **results, pgoff_t *first_index,
unsigned int max_items, int tag);
void f2fs_cache_gang_release(struct f2fs_cached_block **entries,
unsigned int nr_entries);
void f2fs_cache_wait_on_all_writeback(struct f2fs_cached_block_list *cache);
void f2fs_cache_wait_writeback_cond(struct f2fs_cached_block *entry,
enum page_type type);
void f2fs_cache_wait_writeback(struct f2fs_cached_block *entry);
void f2fs_cache_update_tag(struct f2fs_cached_block *entry,
unsigned int clear_from, unsigned int set_to);
struct f2fs_cached_block *f2fs_grab_cache(struct f2fs_cached_block_list *cache,
unsigned long index, int flags);
void f2fs_truncate_locked_cache(struct f2fs_cached_block *entry,
bool drop_dirty);
void f2fs_truncate_cache(struct f2fs_cached_block *entry, bool drop_dirty);
void f2fs_drop_cache_range(struct f2fs_cached_block_list *cache,
unsigned long start, unsigned long len, bool drop_dirty);
#define META_CACHE(sbi) (&(sbi)->meta_blocks)
#define NODE_CACHE(sbi) (&(sbi)->node_blocks)
#define COMPRESS_CACHE(sbi) (&(sbi)->compress_blocks)
#define f2fs_find_meta_cache(sbi, blkaddr) \
f2fs_find_cache(META_CACHE(sbi), blkaddr, 0)
#define f2fs_invalidate_meta_caches(sbi, start, len) \
f2fs_drop_cache_range(META_CACHE(sbi), start, len, false)
#define f2fs_truncate_meta_caches(sbi, start, len) \
f2fs_drop_cache_range(META_CACHE(sbi), start, len, true)
#define f2fs_grab_node_cache(sbi, blkaddr) \
f2fs_grab_cache(NODE_CACHE(sbi), blkaddr, \
F2FS_CACHE_LOCK_CREATE)
#define f2fs_find_node_cache(sbi, blkaddr) \
f2fs_find_cache(NODE_CACHE(sbi), blkaddr, F2FS_CACHE_ACCESS)
#define f2fs_invalidate_node_cache(sbi, blkaddr) \
f2fs_drop_cache_range(NODE_CACHE(sbi), blkaddr, 1, false)
#define f2fs_truncate_node_caches(sbi, start, len) \
f2fs_drop_cache_range(NODE_CACHE(sbi), start, len, true)
unsigned long f2fs_shrink_cache(struct f2fs_sb_info *sbi,
unsigned long nr_to_scan);
unsigned long f2fs_shrink_meta_cache(struct f2fs_sb_info *sbi,
unsigned long nr_to_scan);
unsigned long f2fs_shrink_node_cache(struct f2fs_sb_info *sbi,
unsigned long nr_to_scan);
#define DEF_DIRTY_CACHE_TIMEOUT 5000
#define MIN_DIRTY_CACHE_TIMEOUT 100
#define MAX_DIRTY_CACHE_TIMEOUT 30000
#define DEF_CACHE_WB_DIRTY_THRESH (8192)
struct f2fs_cache_kthread {
struct task_struct *cache_wb_task;
wait_queue_head_t cache_wb_wq;
unsigned int cache_wb_interval;
unsigned int cache_wb_dirty_threshold;
unsigned int cache_wb_total_threshold;
bool cache_wb_throttled; /* last round made no progress */
};
int f2fs_start_cache_wb_thread(struct f2fs_sb_info *sbi);
void f2fs_stop_cache_wb_thread(struct f2fs_sb_info *sbi);
void f2fs_wake_up_cache_wb(struct f2fs_sb_info *sbi);
#endif /* _LINUX_F2FS_CACHE_H */

568
fs/f2fs/checkpoint.c
View File

File diff suppressed because it is too large Load Diff

195
fs/f2fs/compress.c
View File

@ -807,7 +807,7 @@ void f2fs_end_read_compressed_page(struct folio *folio, bool failed,
struct decompress_io_ctx *dic = folio->private;
struct f2fs_sb_info *sbi = dic->sbi;
dec_page_count(sbi, F2FS_RD_DATA);
dec_cache_count(sbi, F2FS_RD_DATA);
if (failed)
WRITE_ONCE(dic->failed, true);
@ -909,7 +909,7 @@ bool f2fs_sanity_check_cluster(struct dnode_of_data *dn)
}
for (i = 1, count = 1; i < cluster_size; i++, count++) {
block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio,
block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry,
dn->ofs_in_node + i);
/* [COMPR_ADDR, ..., COMPR_ADDR] */
@ -950,7 +950,7 @@ static int __f2fs_get_cluster_blocks(struct inode *inode,
int count, i;
for (i = 0, count = 0; i < cluster_size; i++) {
block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio,
block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry,
dn->ofs_in_node + i);
if (__is_valid_data_blkaddr(blkaddr))
@ -1152,11 +1152,11 @@ static int prepare_compress_overwrite(struct compress_ctx *cc,
goto release_and_retry;
}
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
f2fs_compress_ctx_add_page(cc, folio);
if (!folio_test_uptodate(folio)) {
f2fs_handle_page_eio(sbi, folio, DATA);
f2fs_handle_page_eio(sbi, folio->index, DATA);
release_and_retry:
f2fs_put_rpages(cc);
f2fs_unlock_rpages(cc, i + 1);
@ -1228,6 +1228,7 @@ int f2fs_truncate_partial_cluster(struct inode *inode, u64 from, bool lock)
int i;
int err;
repeat:
err = f2fs_is_compressed_cluster(inode, start_idx);
if (err < 0)
return err;
@ -1239,12 +1240,11 @@ int f2fs_truncate_partial_cluster(struct inode *inode, u64 from, bool lock)
/* truncate compressed cluster */
err = f2fs_prepare_compress_overwrite(inode, &pagep,
start_idx, &fsdata);
/* should not be a normal cluster */
f2fs_bug_on(F2FS_I_SB(inode), err == 0);
if (err <= 0)
if (err < 0)
return err;
else if (err == 0)
/* the cluster became non-compressed one due to race case */
goto repeat;
rpages = fsdata;
@ -1328,7 +1328,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
goto out_unlock_op;
for (i = 0; i < cc->cluster_size; i++) {
if (data_blkaddr(dn.inode, dn.node_folio,
if (data_blkaddr(dn.inode, dn.node_entry,
dn.ofs_in_node + i) == NULL_ADDR)
goto out_put_dnode;
}
@ -1360,10 +1360,10 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
page_folio(cc->rpages[i + 1])->index, cic);
fio.compressed_page = cc->cpages[i];
fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_folio,
fio.old_blkaddr = data_blkaddr(dn.inode, dn.node_entry,
dn.ofs_in_node + i + 1);
/* wait for GCed page writeback via META_MAPPING */
/* wait for GCed page writeback via generic cache */
f2fs_wait_on_block_writeback(inode, fio.old_blkaddr);
}
@ -1477,7 +1477,7 @@ void f2fs_compress_write_end_io(struct bio *bio, struct folio *folio)
f2fs_compress_free_page(page);
if (atomic_dec_return(&cic->pending_pages)) {
dec_page_count(sbi, type);
dec_cache_count(sbi, type);
return;
}
@ -1494,12 +1494,12 @@ void f2fs_compress_write_end_io(struct bio *bio, struct folio *folio)
kmem_cache_free(cic_entry_slab, cic);
/*
* Make sure dec_page_count() is the last access to sbi.
* Make sure dec_cache_count() is the last access to sbi.
* Once it drops the F2FS_WB_CP_DATA counter to zero, the
* unmount thread can proceed to destroy sbi and
* sbi->page_array_slab.
*/
dec_page_count(sbi, type);
dec_cache_count(sbi, type);
}
static int f2fs_write_raw_pages(struct compress_ctx *cc,
@ -1551,7 +1551,7 @@ static int f2fs_write_raw_pages(struct compress_ctx *cc,
if (folio_test_writeback(folio)) {
if (wbc->sync_mode == WB_SYNC_NONE)
goto continue_unlock;
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
}
if (!folio_clear_dirty_for_io(folio))
@ -1883,14 +1883,14 @@ void f2fs_put_folio_dic(struct folio *folio, bool in_task)
unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn,
unsigned int ofs_in_node)
{
bool compressed = data_blkaddr(dn->inode, dn->node_folio,
bool compressed = data_blkaddr(dn->inode, dn->node_entry,
ofs_in_node) == COMPRESS_ADDR;
int i = compressed ? 1 : 0;
block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_folio,
block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_entry,
ofs_in_node + i);
for (i += 1; i < F2FS_I(dn->inode)->i_cluster_size; i++) {
block_t blkaddr = data_blkaddr(dn->inode, dn->node_folio,
block_t blkaddr = data_blkaddr(dn->inode, dn->node_entry,
ofs_in_node + i);
if (!__is_valid_data_blkaddr(blkaddr))
@ -1902,30 +1902,19 @@ unsigned int f2fs_cluster_blocks_are_contiguous(struct dnode_of_data *dn,
return compressed ? i - 1 : i;
}
const struct address_space_operations f2fs_compress_aops = {
.release_folio = f2fs_release_folio,
.invalidate_folio = f2fs_invalidate_folio,
.migrate_folio = filemap_migrate_folio,
};
struct address_space *COMPRESS_MAPPING(struct f2fs_sb_info *sbi)
{
return sbi->compress_inode->i_mapping;
}
void f2fs_invalidate_compress_pages_range(struct f2fs_sb_info *sbi,
block_t blkaddr, unsigned int len)
{
if (!sbi->compress_inode)
if (!test_opt(sbi, COMPRESS_CACHE))
return;
invalidate_mapping_pages(COMPRESS_MAPPING(sbi), blkaddr, blkaddr + len - 1);
f2fs_drop_cache_range(COMPRESS_CACHE(sbi), blkaddr, len, false);
}
static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi,
struct folio *folio, nid_t ino, block_t blkaddr)
{
struct folio *cfolio;
int ret;
struct f2fs_cached_block *entry;
if (!test_opt(sbi, COMPRESS_CACHE))
return;
@ -1933,52 +1922,66 @@ static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi,
if (!f2fs_is_valid_blkaddr(sbi, blkaddr, DATA_GENERIC_ENHANCE_READ))
return;
if (!f2fs_available_free_memory(sbi, COMPRESS_PAGE))
if (!f2fs_available_free_memory(sbi, COMPRESS_BLOCK))
return;
cfolio = filemap_get_folio(COMPRESS_MAPPING(sbi), blkaddr);
if (!IS_ERR(cfolio)) {
f2fs_folio_put(cfolio, false);
entry = f2fs_find_cache(COMPRESS_CACHE(sbi), blkaddr,
F2FS_CACHE_ACCESS);
if (!IS_ERR(entry)) {
f2fs_put_cache(entry, false);
return;
}
cfolio = filemap_alloc_folio(__GFP_NOWARN | __GFP_IO, 0, NULL);
if (!cfolio)
entry = f2fs_grab_cache(COMPRESS_CACHE(sbi), blkaddr,
F2FS_CACHE_LOCK_CREATE);
if (IS_ERR(entry))
return;
ret = filemap_add_folio(COMPRESS_MAPPING(sbi), cfolio,
blkaddr, GFP_NOFS);
if (ret) {
f2fs_folio_put(cfolio, false);
/* someone else has populated the entry after f2fs_find_cache() */
if (f2fs_cache_test_uptodate(entry)) {
/*
* a block address can only be owned by one inode, if the
* entry was cached by another inode, one of copies must be
* stale, drop it and let readers load the block from disk.
*/
if (unlikely(entry->ino != ino)) {
f2fs_bug_on(sbi, 1);
f2fs_warn_ratelimited(sbi,
"%s: blkaddr %u was cached by ino %u, current ino %u",
__func__, blkaddr, entry->ino, ino);
f2fs_truncate_locked_cache(entry, false);
}
f2fs_put_cache(entry, true);
return;
}
folio_set_f2fs_data(cfolio, ino);
memcpy(folio_address(cfolio), folio_address(folio), PAGE_SIZE);
folio_mark_uptodate(cfolio);
f2fs_folio_put(cfolio, true);
entry->ino = ino;
memcpy(cache_address(entry), folio_address(folio), sbi->blocksize);
f2fs_cache_set_uptodate(entry);
f2fs_put_cache(entry, true);
}
bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio,
block_t blkaddr)
{
struct folio *cfolio;
struct f2fs_cached_block *entry;
bool hitted = false;
if (!test_opt(sbi, COMPRESS_CACHE))
return false;
cfolio = f2fs_filemap_get_folio(COMPRESS_MAPPING(sbi),
blkaddr, FGP_LOCK | FGP_NOWAIT, GFP_NOFS);
if (!IS_ERR(cfolio)) {
if (folio_test_uptodate(cfolio)) {
entry = f2fs_find_cache(COMPRESS_CACHE(sbi), blkaddr,
F2FS_CACHE_ACCESS);
if (!IS_ERR(entry)) {
f2fs_lock_cache(entry);
if (f2fs_is_compress_cache(entry) &&
f2fs_cache_test_uptodate(entry)) {
atomic_inc(&sbi->compress_page_hit);
memcpy(folio_address(folio),
folio_address(cfolio), folio_size(folio));
cache_address(entry), folio_size(folio));
hitted = true;
}
f2fs_folio_put(cfolio, true);
f2fs_put_cache(entry, true);
}
return hitted;
@ -1986,71 +1989,49 @@ bool f2fs_load_compressed_folio(struct f2fs_sb_info *sbi, struct folio *folio,
void f2fs_invalidate_compress_pages(struct f2fs_sb_info *sbi, nid_t ino)
{
struct address_space *mapping = COMPRESS_MAPPING(sbi);
struct folio_batch fbatch;
pgoff_t index = 0;
pgoff_t end = MAX_BLKADDR(sbi);
struct f2fs_cached_block_list *cache = COMPRESS_CACHE(sbi);
struct f2fs_cached_block *entries[F2FS_ONSTACK_CACHES];
pgoff_t index = 0, end = ULONG_MAX;
int nr;
int i;
if (!mapping->nrpages)
if (!cache->num_entries)
return;
next:
nr = f2fs_cache_gang_lookup(cache, entries, &index, end);
if (!nr)
return;
for (i = 0; i < nr; i++) {
struct f2fs_cached_block *entry = entries[i];
folio_batch_init(&fbatch);
index = entry->index + 1;
do {
unsigned int nr, i;
f2fs_lock_cache(entry);
if (unlikely(!f2fs_is_compress_cache(entry)))
goto unlock;
if (entry->ino != ino)
goto unlock;
nr = filemap_get_folios(mapping, &index, end - 1, &fbatch);
if (!nr)
break;
f2fs_truncate_locked_cache(entry, false);
unlock:
f2fs_unlock_cache(entry);
}
f2fs_cache_gang_release(entries, nr);
for (i = 0; i < nr; i++) {
struct folio *folio = fbatch.folios[i];
folio_lock(folio);
if (folio->mapping != mapping) {
folio_unlock(folio);
continue;
}
if (ino != folio_get_f2fs_data(folio)) {
folio_unlock(folio);
continue;
}
generic_error_remove_folio(mapping, folio);
folio_unlock(folio);
}
folio_batch_release(&fbatch);
if (index < end) {
cond_resched();
} while (index < end);
goto next;
}
}
int f2fs_init_compress_inode(struct f2fs_sb_info *sbi)
void f2fs_init_compress_cache_context(struct f2fs_sb_info *sbi)
{
struct inode *inode;
if (!test_opt(sbi, COMPRESS_CACHE))
return 0;
inode = f2fs_iget(sbi->sb, F2FS_COMPRESS_INO(sbi));
if (IS_ERR(inode))
return PTR_ERR(inode);
sbi->compress_inode = inode;
return;
sbi->compress_percent = COMPRESS_PERCENT;
sbi->compress_watermark = COMPRESS_WATERMARK;
atomic_set(&sbi->compress_page_hit, 0);
return 0;
}
void f2fs_destroy_compress_inode(struct f2fs_sb_info *sbi)
{
if (!sbi->compress_inode)
return;
iput(sbi->compress_inode);
sbi->compress_inode = NULL;
}
int f2fs_init_page_array_cache(struct f2fs_sb_info *sbi)

661
fs/f2fs/data.c
View File

File diff suppressed because it is too large Load Diff

94
fs/f2fs/debug.c
View File

@ -156,12 +156,12 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->allocated_data_blocks = atomic64_read(&sbi->allocated_data_blocks);
/* validation check of the segment numbers */
si->ndirty_node = get_pages(sbi, F2FS_DIRTY_NODES);
si->ndirty_dent = get_pages(sbi, F2FS_DIRTY_DENTS);
si->ndirty_meta = get_pages(sbi, F2FS_DIRTY_META);
si->ndirty_data = get_pages(sbi, F2FS_DIRTY_DATA);
si->ndirty_qdata = get_pages(sbi, F2FS_DIRTY_QDATA);
si->ndirty_imeta = get_pages(sbi, F2FS_DIRTY_IMETA);
si->ndirty_node = get_nr_caches(sbi, F2FS_DIRTY_NODES);
si->ndirty_dent = get_nr_caches(sbi, F2FS_DIRTY_DENTS);
si->ndirty_meta = get_nr_caches(sbi, F2FS_DIRTY_META);
si->ndirty_data = get_nr_caches(sbi, F2FS_DIRTY_DATA);
si->ndirty_qdata = get_nr_caches(sbi, F2FS_DIRTY_QDATA);
si->ndirty_imeta = get_nr_caches(sbi, F2FS_DIRTY_IMETA);
si->ndirty_dirs = sbi->ndirty_inode[DIR_INODE];
si->ndirty_files = sbi->ndirty_inode[FILE_INODE];
si->ndonate_files = sbi->donate_files;
@ -169,13 +169,13 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->ndirty_all = sbi->ndirty_inode[DIRTY_META];
si->aw_cnt = atomic_read(&sbi->atomic_files);
si->max_aw_cnt = atomic_read(&sbi->max_aw_cnt);
si->nr_dio_read = get_pages(sbi, F2FS_DIO_READ);
si->nr_dio_write = get_pages(sbi, F2FS_DIO_WRITE);
si->nr_wb_cp_data = get_pages(sbi, F2FS_WB_CP_DATA);
si->nr_wb_data = get_pages(sbi, F2FS_WB_DATA);
si->nr_rd_data = get_pages(sbi, F2FS_RD_DATA);
si->nr_rd_node = get_pages(sbi, F2FS_RD_NODE);
si->nr_rd_meta = get_pages(sbi, F2FS_RD_META);
si->nr_dio_read = get_nr_caches(sbi, F2FS_DIO_READ);
si->nr_dio_write = get_nr_caches(sbi, F2FS_DIO_WRITE);
si->nr_wb_cp_data = get_nr_caches(sbi, F2FS_WB_CP_DATA);
si->nr_wb_data = get_nr_caches(sbi, F2FS_WB_DATA);
si->nr_rd_data = get_nr_caches(sbi, F2FS_RD_DATA);
si->nr_rd_node = get_nr_caches(sbi, F2FS_RD_NODE);
si->nr_rd_meta = get_nr_caches(sbi, F2FS_RD_META);
if (SM_I(sbi)->fcc_info) {
si->nr_flushed =
atomic_read(&SM_I(sbi)->fcc_info->issued_flush);
@ -222,13 +222,11 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->free_secs = free_sections(sbi);
si->prefree_count = prefree_segments(sbi);
si->dirty_count = dirty_segments(sbi);
if (sbi->node_inode)
si->node_pages = NODE_MAPPING(sbi)->nrpages;
if (sbi->meta_inode)
si->meta_pages = META_MAPPING(sbi)->nrpages;
si->node_caches = NODE_CACHE(sbi)->num_entries;
si->meta_caches = META_CACHE(sbi)->num_entries;
#ifdef CONFIG_F2FS_FS_COMPRESSION
if (sbi->compress_inode) {
si->compress_pages = COMPRESS_MAPPING(sbi)->nrpages;
if (test_opt(sbi, COMPRESS_CACHE)) {
si->compress_pages = COMPRESS_CACHE(sbi)->num_entries;
si->compress_page_hit = atomic_read(&sbi->compress_page_hit);
}
#endif
@ -343,9 +341,9 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
/* build nm */
si->base_mem += sizeof(struct f2fs_nm_info);
si->base_mem += __bitmap_size(sbi, NAT_BITMAP);
si->base_mem += F2FS_BLK_TO_BYTES(NM_I(sbi)->nat_bits_blocks);
si->base_mem += F2FS_BLK_TO_BYTES(sbi, NM_I(sbi)->nat_bits_blocks);
si->base_mem += NM_I(sbi)->nat_blocks *
f2fs_bitmap_size(NAT_ENTRY_PER_BLOCK);
f2fs_bitmap_size(NAT_ENTRY_PER_BLOCK(sbi));
si->base_mem += NM_I(sbi)->nat_blocks / 8;
si->base_mem += NM_I(sbi)->nat_blocks * sizeof(unsigned short);
@ -382,22 +380,36 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
si->cache_mem += si->ext_mem[i];
}
si->page_mem = 0;
if (sbi->node_inode) {
unsigned long npages = NODE_MAPPING(sbi)->nrpages;
si->cache_entry_mem[F2FS_META_CACHE] =
(unsigned long long)META_CACHE(sbi)->num_entries *
sizeof(struct f2fs_cached_block);
si->cache_data_mem[F2FS_META_CACHE] =
(unsigned long long)META_CACHE(sbi)->num_entries * sbi->blocksize;
si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
}
if (sbi->meta_inode) {
unsigned long npages = META_MAPPING(sbi)->nrpages;
si->cache_entry_mem[F2FS_NODE_CACHE] =
(unsigned long long)NODE_CACHE(sbi)->num_entries *
sizeof(struct f2fs_cached_block);
si->cache_data_mem[F2FS_NODE_CACHE] =
(unsigned long long)NODE_CACHE(sbi)->num_entries * sbi->blocksize;
si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
}
si->cache_mem += si->cache_entry_mem[F2FS_META_CACHE] +
si->cache_entry_mem[F2FS_NODE_CACHE];
si->page_mem = si->cache_data_mem[F2FS_META_CACHE] +
si->cache_data_mem[F2FS_NODE_CACHE];
#ifdef CONFIG_F2FS_FS_COMPRESSION
if (sbi->compress_inode) {
unsigned long npages = COMPRESS_MAPPING(sbi)->nrpages;
if (test_opt(sbi, COMPRESS_CACHE)) {
si->cache_entry_mem[F2FS_COMPRESS_CACHE] =
(unsigned long long)COMPRESS_CACHE(sbi)->num_entries *
sizeof(struct f2fs_cached_block);
si->cache_data_mem[F2FS_COMPRESS_CACHE] =
(unsigned long long)COMPRESS_CACHE(sbi)->num_entries *
sbi->blocksize;
si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
si->cache_mem += si->cache_entry_mem[F2FS_COMPRESS_CACHE];
si->page_mem += si->cache_data_mem[F2FS_COMPRESS_CACHE];
} else {
si->cache_entry_mem[F2FS_COMPRESS_CACHE] = 0;
si->cache_data_mem[F2FS_COMPRESS_CACHE] = 0;
}
#endif
}
@ -700,7 +712,7 @@ static int stat_show(struct seq_file *s, void *v)
si->aw_cnt, si->max_aw_cnt);
seq_printf(s, " - compress: %4d, hit:%8d\n", si->compress_pages, si->compress_page_hit);
seq_printf(s, " - nodes: %4d in %4d\n",
si->ndirty_node, si->node_pages);
si->ndirty_node, si->node_caches);
seq_printf(s, " - dents: %4d in dirs:%4d (%4d)\n",
si->ndirty_dent, si->ndirty_dirs, si->ndirty_all);
seq_printf(s, " - data: %4d in files:%4d\n",
@ -708,7 +720,7 @@ static int stat_show(struct seq_file *s, void *v)
seq_printf(s, " - quota data: %4d in quota files:%4d\n",
si->ndirty_qdata, si->nquota_files);
seq_printf(s, " - meta: %4d in %4d\n",
si->ndirty_meta, si->meta_pages);
si->ndirty_meta, si->meta_caches);
seq_printf(s, " - imeta: %4d\n",
si->ndirty_imeta);
seq_printf(s, " - fsync mark: %4lld\n",
@ -756,6 +768,18 @@ static int stat_show(struct seq_file *s, void *v)
si->ext_mem[EX_READ] >> 10);
seq_printf(s, " - block age extent cache: %llu KB\n",
si->ext_mem[EX_BLOCK_AGE] >> 10);
seq_printf(s, " - meta entry: %llu KB, meta cache: %llu KB\n",
si->cache_entry_mem[F2FS_META_CACHE] >> 10,
si->cache_data_mem[F2FS_META_CACHE] >> 10);
seq_printf(s, " - node entry: %llu KB, node cache: %llu KB\n",
si->cache_entry_mem[F2FS_NODE_CACHE] >> 10,
si->cache_data_mem[F2FS_NODE_CACHE] >> 10);
#ifdef CONFIG_F2FS_FS_COMPRESSION
if (test_opt(sbi, COMPRESS_CACHE))
seq_printf(s, " - compress entry: %llu KB, compress cache: %llu KB\n",
si->cache_entry_mem[F2FS_COMPRESS_CACHE] >> 10,
si->cache_data_mem[F2FS_COMPRESS_CACHE] >> 10);
#endif
seq_printf(s, " - paged : %llu KB\n",
si->page_mem >> 10);
}

221
fs/f2fs/dir.c
View File

@ -195,7 +195,7 @@ static struct f2fs_dir_entry *find_in_block(struct inode *dir,
int *max_slots,
bool use_hash)
{
struct f2fs_dentry_block *dentry_blk;
void *dentry_blk;
struct f2fs_dentry_ptr d;
dentry_blk = folio_address(dentry_folio);
@ -282,7 +282,7 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(const struct f2fs_dentry_ptr *d,
static struct f2fs_dir_entry *find_in_level(struct inode *dir,
unsigned int level,
const struct f2fs_filename *fname,
struct folio **res_folio,
void **dentry_block,
bool use_hash)
{
int s = GET_DENTRY_SLOTS(fname->disk_name.len);
@ -313,7 +313,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
bidx = next_pgofs;
continue;
} else {
*res_folio = dentry_folio;
*dentry_block = dentry_folio;
break;
}
}
@ -321,11 +321,11 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
de = find_in_block(dir, dentry_folio, fname, &max_slots, use_hash);
if (IS_ERR(de)) {
f2fs_folio_put(dentry_folio, false);
*res_folio = ERR_CAST(de);
*dentry_block = ERR_CAST(de);
de = NULL;
break;
} else if (de) {
*res_folio = dentry_folio;
*dentry_block = dentry_folio;
break;
}
@ -352,7 +352,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir,
const struct f2fs_filename *fname,
struct folio **res_folio)
void **dentry_block)
{
unsigned long npages = dir_blocks(dir);
struct f2fs_dir_entry *de = NULL;
@ -360,13 +360,13 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir,
unsigned int level;
bool use_hash = true;
*res_folio = NULL;
*dentry_block = NULL;
#if IS_ENABLED(CONFIG_UNICODE)
start_find_entry:
#endif
if (f2fs_has_inline_dentry(dir)) {
de = f2fs_find_in_inline_dir(dir, fname, res_folio, use_hash);
de = f2fs_find_in_inline_dir(dir, fname, dentry_block, use_hash);
goto out;
}
@ -382,8 +382,8 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir,
}
for (level = 0; level < max_depth; level++) {
de = find_in_level(dir, level, fname, res_folio, use_hash);
if (de || IS_ERR(*res_folio))
de = find_in_level(dir, level, fname, dentry_block, use_hash);
if (de || IS_ERR(*dentry_block))
break;
}
@ -408,7 +408,7 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir,
* Entry is guaranteed to be valid.
*/
struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir,
const struct qstr *child, struct folio **res_folio)
const struct qstr *child, void **dentry_block)
{
struct f2fs_dir_entry *de = NULL;
struct f2fs_filename fname;
@ -417,67 +417,78 @@ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir,
err = f2fs_setup_filename(dir, child, 1, &fname);
if (err) {
if (err == -ENOENT)
*res_folio = NULL;
*dentry_block = NULL;
else
*res_folio = ERR_PTR(err);
*dentry_block = ERR_PTR(err);
return NULL;
}
de = __f2fs_find_entry(dir, &fname, res_folio);
de = __f2fs_find_entry(dir, &fname, dentry_block);
f2fs_free_filename(&fname);
return de;
}
struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, struct folio **f)
struct f2fs_dir_entry *f2fs_parent_dir(struct inode *dir, void **dentry_block)
{
return f2fs_find_entry(dir, &dotdot_name, f);
return f2fs_find_entry(dir, &dotdot_name, dentry_block);
}
ino_t f2fs_inode_by_name(struct inode *dir, const struct qstr *qstr,
struct folio **folio)
void **dentry_block)
{
ino_t res = 0;
struct f2fs_dir_entry *de;
de = f2fs_find_entry(dir, qstr, folio);
de = f2fs_find_entry(dir, qstr, dentry_block);
if (de) {
res = le32_to_cpu(de->ino);
f2fs_folio_put(*folio, false);
f2fs_put_dentry_block(*dentry_block, false);
}
return res;
}
void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de,
struct folio *folio, struct inode *inode)
void *dentry_block, struct inode *inode)
{
enum page_type type = f2fs_has_inline_dentry(dir) ? NODE : DATA;
if (f2fs_dentry_is_cache(dentry_block)) {
struct f2fs_cached_block *entry =
f2fs_dentry_cache(dentry_block);
folio_lock(folio);
f2fs_folio_wait_writeback(folio, type, true, true);
de->ino = cpu_to_le32(inode->i_ino);
de->file_type = fs_umode_to_ftype(inode->i_mode);
folio_mark_dirty(folio);
f2fs_lock_cache(entry);
f2fs_cache_wait_writeback(entry);
de->ino = cpu_to_le32(inode->i_ino);
de->file_type = fs_umode_to_ftype(inode->i_mode);
f2fs_mark_cache_dirty(entry);
} else {
struct folio *folio = f2fs_dentry_folio(dentry_block);
folio_lock(folio);
f2fs_folio_wait_writeback(folio, true, true);
de->ino = cpu_to_le32(inode->i_ino);
de->file_type = fs_umode_to_ftype(inode->i_mode);
folio_mark_dirty(folio);
}
inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
f2fs_mark_inode_dirty_sync(dir, true);
f2fs_folio_put(folio, true);
f2fs_put_dentry_block(dentry_block, true);
}
static void init_dent_inode(struct inode *dir, struct inode *inode,
const struct f2fs_filename *fname,
struct folio *ifolio)
struct f2fs_cached_block *ientry)
{
struct f2fs_inode *ri;
if (!fname) /* tmpfile case? */
return;
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_cache_wait_writeback(ientry);
/* copy name info. to this inode folio */
ri = F2FS_INODE(ifolio);
ri = F2FS_INODE(ientry);
ri->i_namelen = cpu_to_le32(fname->disk_name.len);
memcpy(ri->i_name, fname->disk_name.name, fname->disk_name.len);
if (IS_ENCRYPTED(dir)) {
@ -498,7 +509,7 @@ static void init_dent_inode(struct inode *dir, struct inode *inode,
file_lost_pino(inode);
}
}
folio_mark_dirty(ifolio);
f2fs_mark_cache_dirty(ientry);
}
void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent,
@ -515,22 +526,22 @@ void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent,
}
static int make_empty_dir(struct inode *inode,
struct inode *parent, struct folio *folio)
struct inode *parent, struct f2fs_cached_block *ientry)
{
struct folio *dentry_folio;
struct f2fs_dentry_block *dentry_blk;
void *dentry_blk;
struct f2fs_dentry_ptr d;
if (f2fs_has_inline_dentry(inode))
return f2fs_make_empty_inline_dir(inode, parent, folio);
return f2fs_make_empty_inline_dir(inode, parent, ientry);
dentry_folio = f2fs_get_new_data_folio(inode, folio, 0, true);
dentry_folio = f2fs_get_new_data_folio(inode, ientry, 0, true);
if (IS_ERR(dentry_folio))
return PTR_ERR(dentry_folio);
dentry_blk = folio_address(dentry_folio);
make_dentry_ptr_block(NULL, &d, dentry_blk);
make_dentry_ptr_block(inode, &d, dentry_blk);
f2fs_do_make_empty_dir(inode, parent, &d);
folio_mark_dirty(dentry_folio);
@ -538,50 +549,50 @@ static int make_empty_dir(struct inode *inode,
return 0;
}
struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir,
const struct f2fs_filename *fname, struct folio *dfolio)
struct f2fs_cached_block *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir,
const struct f2fs_filename *fname, struct f2fs_cached_block *dentry)
{
struct folio *folio;
struct f2fs_cached_block *ientry;
int err;
if (is_inode_flag_set(inode, FI_NEW_INODE)) {
folio = f2fs_new_inode_folio(inode);
if (IS_ERR(folio))
return folio;
ientry = f2fs_new_inode_cache(inode);
if (IS_ERR(ientry))
return ientry;
if (S_ISDIR(inode->i_mode)) {
/* in order to handle error case */
folio_get(folio);
err = make_empty_dir(inode, dir, folio);
f2fs_cache_get(ientry);
err = make_empty_dir(inode, dir, ientry);
if (err) {
folio_lock(folio);
f2fs_lock_cache(ientry);
goto put_error;
}
folio_put(folio);
f2fs_put_cache(ientry, false);
}
err = f2fs_init_acl(inode, dir, folio, dfolio);
err = f2fs_init_acl(inode, dir, ientry, dentry);
if (err)
goto put_error;
err = f2fs_init_security(inode, dir,
fname ? fname->usr_fname : NULL,
folio);
ientry);
if (err)
goto put_error;
if (IS_ENCRYPTED(inode)) {
err = fscrypt_set_context(inode, folio);
err = fscrypt_set_context(inode, ientry);
if (err)
goto put_error;
}
} else {
folio = f2fs_get_inode_folio(F2FS_I_SB(dir), inode->i_ino);
if (IS_ERR(folio))
return folio;
ientry = f2fs_get_inode_cache(F2FS_I_SB(dir), inode->i_ino);
if (IS_ERR(ientry))
return ientry;
}
init_dent_inode(dir, inode, fname, folio);
init_dent_inode(dir, inode, fname, ientry);
/*
* This file should be checkpointed during fsync.
@ -598,12 +609,12 @@ struct folio *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir,
f2fs_remove_orphan_inode(F2FS_I_SB(dir), inode->i_ino);
f2fs_i_links_write(inode, true);
}
return folio;
return ientry;
put_error:
clear_nlink(inode);
f2fs_update_inode(inode, folio);
f2fs_folio_put(folio, true);
f2fs_update_inode(inode, ientry);
f2fs_put_cache(ientry, true);
return ERR_PTR(err);
}
@ -645,14 +656,14 @@ int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots)
goto next;
}
bool f2fs_has_enough_room(struct inode *dir, struct folio *ifolio,
bool f2fs_has_enough_room(struct inode *dir, struct f2fs_cached_block *ientry,
const struct f2fs_filename *fname)
{
struct f2fs_dentry_ptr d;
unsigned int bit_pos;
int slots = GET_DENTRY_SLOTS(fname->disk_name.len);
make_dentry_ptr_inline(dir, &d, inline_data_addr(dir, ifolio));
make_dentry_ptr_inline(dir, &d, inline_data_addr(dir, ientry));
bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max);
@ -690,9 +701,9 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname,
unsigned long bidx, block;
unsigned int nbucket, nblock;
struct folio *dentry_folio = NULL;
struct f2fs_dentry_block *dentry_blk = NULL;
void *dentry_blk = NULL;
struct f2fs_dentry_ptr d;
struct folio *folio = NULL;
struct f2fs_cached_block *entry = NULL;
int slots, err = 0;
level = 0;
@ -727,9 +738,9 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname,
return PTR_ERR(dentry_folio);
dentry_blk = folio_address(dentry_folio);
bit_pos = f2fs_room_for_filename(&dentry_blk->dentry_bitmap,
slots, NR_DENTRY_IN_BLOCK);
if (bit_pos < NR_DENTRY_IN_BLOCK)
make_dentry_ptr_block(dir, &d, dentry_blk);
bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max);
if (bit_pos < d.max)
goto add_dentry;
f2fs_folio_put(dentry_folio, true);
@ -739,18 +750,17 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname,
++level;
goto start;
add_dentry:
f2fs_folio_wait_writeback(dentry_folio, DATA, true, true);
f2fs_folio_wait_writeback(dentry_folio, true, true);
if (inode) {
f2fs_down_write(&F2FS_I(inode)->i_sem);
folio = f2fs_init_inode_metadata(inode, dir, fname, NULL);
if (IS_ERR(folio)) {
err = PTR_ERR(folio);
entry = f2fs_init_inode_metadata(inode, dir, fname, NULL);
if (IS_ERR(entry)) {
err = PTR_ERR(entry);
goto fail;
}
}
make_dentry_ptr_block(NULL, &d, dentry_blk);
f2fs_update_dentry(ino, mode, &d, &fname->disk_name, fname->hash,
bit_pos);
@ -761,9 +771,9 @@ int f2fs_add_regular_entry(struct inode *dir, const struct f2fs_filename *fname,
/* synchronize inode page's data from inode cache */
if (is_inode_flag_set(inode, FI_NEW_INODE))
f2fs_update_inode(inode, folio);
f2fs_update_inode(inode, entry);
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
}
f2fs_update_parent_metadata(dir, inode, current_depth);
@ -805,7 +815,7 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name,
struct inode *inode, nid_t ino, umode_t mode)
{
struct f2fs_filename fname;
struct folio *folio = NULL;
void *dentry_blk = NULL;
struct f2fs_dir_entry *de = NULL;
int err;
@ -821,14 +831,14 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name,
* consistency more.
*/
if (current != F2FS_I(dir)->task) {
de = __f2fs_find_entry(dir, &fname, &folio);
de = __f2fs_find_entry(dir, &fname, &dentry_blk);
F2FS_I(dir)->task = NULL;
}
if (de) {
f2fs_folio_put(folio, false);
f2fs_put_dentry_block(dentry_blk, false);
err = -EEXIST;
} else if (IS_ERR(folio)) {
err = PTR_ERR(folio);
} else if (IS_ERR(dentry_blk)) {
err = PTR_ERR(dentry_blk);
} else {
err = f2fs_add_dentry(dir, &fname, inode, ino, mode);
}
@ -839,16 +849,16 @@ int f2fs_do_add_link(struct inode *dir, const struct qstr *name,
int f2fs_do_tmpfile(struct inode *inode, struct inode *dir,
struct f2fs_filename *fname)
{
struct folio *folio;
struct f2fs_cached_block *ientry;
int err = 0;
f2fs_down_write(&F2FS_I(inode)->i_sem);
folio = f2fs_init_inode_metadata(inode, dir, fname, NULL);
if (IS_ERR(folio)) {
err = PTR_ERR(folio);
ientry = f2fs_init_inode_metadata(inode, dir, fname, NULL);
if (IS_ERR(ientry)) {
err = PTR_ERR(ientry);
goto fail;
}
f2fs_folio_put(folio, true);
f2fs_put_cache(ientry, true);
clear_inode_flag(inode, FI_NEW_INODE);
f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
@ -884,13 +894,15 @@ void f2fs_drop_nlink(struct inode *dir, struct inode *inode)
* It only removes the dentry from the dentry page, corresponding name
* entry in name page does not need to be touched during deletion.
*/
void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio,
void f2fs_delete_entry(struct f2fs_dir_entry *dentry, void *dentry_block,
struct inode *dir, struct inode *inode)
{
struct f2fs_dentry_block *dentry_blk;
void *dentry_blk;
struct f2fs_dentry_ptr d;
struct folio *folio;
unsigned int bit_pos;
int slots = GET_DENTRY_SLOTS(le16_to_cpu(dentry->name_len));
pgoff_t index = folio->index;
pgoff_t index;
int i;
f2fs_update_time(F2FS_I_SB(dir), REQ_TIME);
@ -899,24 +911,27 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio,
f2fs_add_ino_entry(F2FS_I_SB(dir), dir->i_ino, TRANS_DIR_INO);
if (f2fs_has_inline_dentry(dir))
return f2fs_delete_inline_entry(dentry, folio, dir, inode);
return f2fs_delete_inline_entry(dentry,
f2fs_dentry_cache(dentry_block), dir, inode);
folio = f2fs_dentry_folio(dentry_block);
index = folio->index;
folio_lock(folio);
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
dentry_blk = folio_address(folio);
bit_pos = dentry - dentry_blk->dentry;
make_dentry_ptr_block(dir, &d, dentry_blk);
bit_pos = dentry - d.dentry;
for (i = 0; i < slots; i++)
__clear_bit_le(bit_pos + i, &dentry_blk->dentry_bitmap);
__clear_bit_le(bit_pos + i, d.bitmap);
/* Let's check and deallocate this dentry page */
bit_pos = find_next_bit_le(&dentry_blk->dentry_bitmap,
NR_DENTRY_IN_BLOCK,
0);
bit_pos = find_next_bit_le(d.bitmap, d.max, 0);
folio_mark_dirty(folio);
if (bit_pos == NR_DENTRY_IN_BLOCK &&
!f2fs_truncate_hole(dir, index, index + 1)) {
if (bit_pos == d.max &&
!f2fs_truncate_hole(dir, index, index + 1)) {
f2fs_clear_page_cache_dirty_tag(folio);
folio_clear_dirty_for_io(folio);
folio_clear_uptodate(folio);
@ -938,7 +953,8 @@ bool f2fs_empty_dir(struct inode *dir)
{
unsigned long bidx = 0;
unsigned int bit_pos;
struct f2fs_dentry_block *dentry_blk;
void *dentry_blk;
struct f2fs_dentry_ptr d;
unsigned long nblock = dir_blocks(dir);
if (f2fs_has_inline_dentry(dir))
@ -959,17 +975,16 @@ bool f2fs_empty_dir(struct inode *dir)
}
dentry_blk = folio_address(dentry_folio);
make_dentry_ptr_block(dir, &d, dentry_blk);
if (bidx == 0)
bit_pos = 2;
else
bit_pos = 0;
bit_pos = find_next_bit_le(&dentry_blk->dentry_bitmap,
NR_DENTRY_IN_BLOCK,
bit_pos);
bit_pos = find_next_bit_le(d.bitmap, d.max, bit_pos);
f2fs_folio_put(dentry_folio, false);
if (bit_pos < NR_DENTRY_IN_BLOCK)
if (bit_pos < d.max)
return false;
bidx++;
@ -1051,7 +1066,7 @@ int f2fs_fill_dentries(struct dir_context *ctx, struct f2fs_dentry_ptr *d,
}
if (readdir_ra)
f2fs_ra_node_page(sbi, le32_to_cpu(de->ino));
f2fs_ra_node_cache(sbi, le32_to_cpu(de->ino));
ctx->pos = start_pos + bit_pos;
found_valid_dirent = true;
@ -1066,10 +1081,12 @@ static int f2fs_readdir(struct file *file, struct dir_context *ctx)
{
struct inode *inode = file_inode(file);
unsigned long npages = dir_blocks(inode);
struct f2fs_dentry_block *dentry_blk = NULL;
void *dentry_blk = NULL;
struct file_ra_state *ra = &file->f_ra;
loff_t start_pos = ctx->pos;
unsigned int n = ((unsigned long)ctx->pos / NR_DENTRY_IN_BLOCK);
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
unsigned int entries = sbi->dentries_per_block;
unsigned int n = (unsigned long)ctx->pos / entries;
struct f2fs_dentry_ptr d;
struct fscrypt_str fstr = FSTR_INIT(NULL, 0);
int err = 0;
@ -1089,7 +1106,7 @@ static int f2fs_readdir(struct file *file, struct dir_context *ctx)
goto out_free;
}
for (; n < npages; ctx->pos = n * NR_DENTRY_IN_BLOCK) {
for (; n < npages; ctx->pos = n * entries) {
struct folio *dentry_folio;
pgoff_t next_pgofs;
@ -1122,7 +1139,7 @@ static int f2fs_readdir(struct file *file, struct dir_context *ctx)
make_dentry_ptr_block(inode, &d, dentry_blk);
err = f2fs_fill_dentries(ctx, &d,
n * NR_DENTRY_IN_BLOCK, &fstr);
n * entries, &fstr);
f2fs_folio_put(dentry_folio, false);
if (err)
break;

25
fs/f2fs/extent_cache.c
View File

@ -20,10 +20,10 @@
#include "segment.h"
#include <trace/events/f2fs.h>
bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio)
bool sanity_check_extent_cache(struct inode *inode, struct f2fs_cached_block *ientry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_extent *i_ext = &F2FS_INODE(ifolio)->i_ext;
struct f2fs_extent *i_ext = &F2FS_INODE(ientry)->i_ext;
struct extent_info ei;
int devi;
@ -416,11 +416,11 @@ static void __drop_largest_extent(struct extent_tree *et,
}
}
void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio)
void f2fs_init_read_extent_tree(struct inode *inode, struct f2fs_cached_block *ientry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct extent_tree_info *eti = &sbi->extent_tree[EX_READ];
struct f2fs_extent *i_ext = &F2FS_INODE(ifolio)->i_ext;
struct f2fs_extent *i_ext = &F2FS_INODE(ientry)->i_ext;
struct extent_tree *et;
struct extent_node *en;
struct extent_info ei = {0};
@ -428,9 +428,9 @@ void f2fs_init_read_extent_tree(struct inode *inode, struct folio *ifolio)
if (!__may_extent_tree(inode, EX_READ)) {
/* drop largest read extent */
if (i_ext->len) {
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_cache_wait_writeback(ientry);
i_ext->len = 0;
folio_mark_dirty(ifolio);
f2fs_mark_cache_dirty(ientry);
}
set_inode_flag(inode, FI_NO_EXTENT);
return;
@ -904,11 +904,12 @@ static int __get_new_block_age(struct inode *inode, struct extent_info *ei,
struct extent_info tei = *ei; /* only fofs and len are valid */
/*
* When I/O is not aligned to a PAGE_SIZE, update will happen to the last
* file block even in seq write. So don't record age for newly last file
* block here.
* When I/O is not aligned to the filesystem block size, update will
* happen to the last file block even in seq write. Do not record the age
* of the new last file block here.
*/
if ((f_size >> PAGE_SHIFT) == ei->fofs && f_size & (PAGE_SIZE - 1) &&
if ((f_size >> sbi->log_blocksize) == ei->fofs &&
f_size & F2FS_BLKSIZE_MASK(sbi) &&
blkaddr == NEW_ADDR)
return -EINVAL;
@ -956,8 +957,8 @@ static void __update_extent_cache(struct dnode_of_data *dn, enum extent_type typ
if (!__may_extent_tree(dn->inode, type))
return;
ei.fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_folio), dn->inode) +
dn->ofs_in_node;
ei.fofs = f2fs_start_bidx_of_node(ofs_of_node(F2FS_I_SB(dn->inode),
dn->node_entry), dn->inode) + dn->ofs_in_node;
ei.len = 1;
if (type == EX_READ) {

518
fs/f2fs/f2fs.h
View File

File diff suppressed because it is too large Load Diff

241
fs/f2fs/file.c
View File

@ -110,7 +110,8 @@ static vm_fault_t f2fs_filemap_fault(struct vm_fault *vmf)
ret = filemap_fault(vmf);
if (ret & VM_FAULT_LOCKED)
f2fs_update_iostat(F2FS_I_SB(inode), inode,
APP_MAPPED_READ_IO, F2FS_BLKSIZE);
APP_MAPPED_READ_IO,
F2FS_BLKSIZE(F2FS_I_SB(inode)));
trace_f2fs_filemap_fault(inode, vmf->pgoff, flags, ret);
@ -124,6 +125,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf)
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct dnode_of_data dn;
bool need_alloc = !f2fs_is_pinned_file(inode);
loff_t isize;
int err = 0;
vm_fault_t ret;
@ -186,9 +188,26 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf)
file_update_time(vmf->vma->vm_file);
filemap_invalidate_lock_shared(inode->i_mapping);
/*
* mmap stores bypass the COW inode and would break atomicity. Check
* under invalidate_lock: f2fs_ioc_start_atomic_write() holds it while
* flushing dirty folios and setting FI_ATOMIC_FILE, so either it sees
* our dirty folio and writes it back, or we see the flag.
*/
if (unlikely(f2fs_is_atomic_file(inode))) {
err = -EOPNOTSUPP;
goto out_sem;
}
folio_lock(folio);
/*
* A concurrent buffered write can extend isize, so we store isize in a
* local variable to let the zeroing offset match the EOF check below.
*/
isize = i_size_read(inode);
if (unlikely(folio->mapping != inode->i_mapping ||
folio_pos(folio) > i_size_read(inode) ||
folio_pos(folio) > isize ||
!folio_test_uptodate(folio))) {
folio_unlock(folio);
err = -EFAULT;
@ -212,28 +231,21 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf)
goto out_sem;
}
f2fs_folio_wait_writeback(folio, DATA, false, true);
f2fs_folio_wait_writeback(folio, false, true);
/* wait for GCed page writeback via META_MAPPING */
/* wait for GCed page writeback via generic cache */
f2fs_wait_on_block_writeback(inode, dn.data_blkaddr);
/*
* check to see if the page is mapped already (no holes)
*/
if (folio_test_mappedtodisk(folio))
goto out_sem;
/* page is wholly or partially inside EOF */
if (((loff_t)(folio->index + 1) << PAGE_SHIFT) >
i_size_read(inode)) {
if (((loff_t)(folio->index + 1) << PAGE_SHIFT) > isize) {
loff_t offset;
offset = i_size_read(inode) & ~PAGE_MASK;
offset = isize & ~PAGE_MASK;
folio_zero_segment(folio, offset, folio_size(folio));
}
folio_mark_dirty(folio);
f2fs_update_iostat(sbi, inode, APP_MAPPED_IO, F2FS_BLKSIZE);
f2fs_update_iostat(sbi, inode, APP_MAPPED_IO, F2FS_BLKSIZE(sbi));
f2fs_update_time(sbi, REQ_TIME);
out_sem:
@ -307,13 +319,13 @@ static inline enum cp_reason_type need_do_checkpoint(struct inode *inode)
static bool need_inode_page_update(struct f2fs_sb_info *sbi, nid_t ino)
{
struct folio *i = filemap_get_folio(NODE_MAPPING(sbi), ino);
struct f2fs_cached_block *entry = f2fs_find_node_cache(sbi, ino);
bool ret = false;
/* But we need to avoid that there are some inode updates */
if ((!IS_ERR(i) && folio_test_dirty(i)) ||
if ((!IS_ERR(entry) && f2fs_cache_test_dirty(entry)) ||
f2fs_need_inode_block_update(sbi, ino))
ret = true;
f2fs_folio_put(i, false);
f2fs_put_cache(entry, false);
return ret;
}
@ -339,10 +351,6 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
nid_t ino = inode->i_ino;
int ret = 0;
enum cp_reason_type cp_reason = 0;
struct writeback_control wbc = {
.sync_mode = WB_SYNC_ALL,
.nr_to_write = LONG_MAX,
};
unsigned int seq_id = 0;
if (unlikely(f2fs_readonly(inode->i_sb)))
@ -421,7 +429,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
}
sync_nodes:
atomic_inc(&sbi->wb_sync_req[NODE]);
ret = f2fs_fsync_node_pages(sbi, inode, &wbc, atomic, &seq_id);
ret = f2fs_fsync_node_caches(sbi, inode, atomic, &seq_id);
atomic_dec(&sbi->wb_sync_req[NODE]);
if (ret)
goto out;
@ -447,7 +455,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
* given fsync mark.
*/
if (!atomic) {
ret = f2fs_wait_on_node_pages_writeback(sbi, seq_id);
ret = f2fs_wait_on_node_caches_writeback(sbi, seq_id);
if (ret)
goto out;
}
@ -484,7 +492,7 @@ static bool __found_offset(struct address_space *mapping,
bool compressed_cluster = false;
if (f2fs_compressed_file(inode)) {
block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_folio,
block_t first_blkaddr = data_blkaddr(dn->inode, dn->node_entry,
ALIGN_DOWN(dn->ofs_in_node, F2FS_I(inode)->i_cluster_size));
compressed_cluster = first_blkaddr == COMPRESS_ADDR;
@ -513,7 +521,8 @@ static bool __found_offset(struct address_space *mapping,
static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence)
{
struct inode *inode = file->f_mapping->host;
loff_t maxbytes = F2FS_BLK_TO_BYTES(max_file_blocks(inode));
loff_t maxbytes = F2FS_BLK_TO_BYTES(F2FS_I_SB(inode),
max_file_blocks(F2FS_I_SB(inode), inode));
struct dnode_of_data dn;
pgoff_t pgofs, end_offset;
loff_t data_ofs = offset;
@ -537,9 +546,10 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence)
}
}
pgofs = (pgoff_t)(offset >> PAGE_SHIFT);
pgofs = F2FS_BYTES_TO_BLK(F2FS_I_SB(inode), offset);
for (; data_ofs < isize; data_ofs = (loff_t)pgofs << PAGE_SHIFT) {
for (; data_ofs < isize;
data_ofs = F2FS_BLK_TO_BYTES(F2FS_I_SB(inode), pgofs)) {
set_new_dnode(&dn, inode, NULL, NULL, 0);
err = f2fs_get_dnode_of_data(&dn, pgofs, LOOKUP_NODE);
if (err && err != -ENOENT) {
@ -554,12 +564,12 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence)
}
}
end_offset = ADDRS_PER_PAGE(dn.node_folio, inode);
end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
/* find data/hole in dnode block */
for (; dn.ofs_in_node < end_offset;
dn.ofs_in_node++, pgofs++,
data_ofs = (loff_t)pgofs << PAGE_SHIFT) {
data_ofs = F2FS_BLK_TO_BYTES(F2FS_I_SB(inode), pgofs)) {
block_t blkaddr;
blkaddr = f2fs_data_blkaddr(&dn);
@ -595,7 +605,8 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence)
static loff_t f2fs_llseek(struct file *file, loff_t offset, int whence)
{
struct inode *inode = file->f_mapping->host;
loff_t maxbytes = F2FS_BLK_TO_BYTES(max_file_blocks(inode));
loff_t maxbytes = F2FS_BLK_TO_BYTES(F2FS_I_SB(inode),
max_file_blocks(F2FS_I_SB(inode), inode));
switch (whence) {
case SEEK_SET:
@ -716,7 +727,7 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count)
block_t blkstart;
int blklen = 0;
addr = get_dnode_addr(dn->inode, dn->node_folio) + ofs;
addr = get_dnode_addr(dn->inode, dn->node_entry) + ofs;
blkstart = le32_to_cpu(*addr);
/* Assumption: truncation starts with cluster */
@ -780,7 +791,7 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count)
* once we invalidate valid blkaddr in range [ofs, ofs + count],
* we will invalidate all blkaddr in the whole range.
*/
fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_folio),
fofs = f2fs_start_bidx_of_node(ofs_of_node(sbi, dn->node_entry),
dn->inode) + ofs;
f2fs_update_read_extent_cache_range(dn, fofs, 0, len);
f2fs_update_age_extent_cache_range(dn, fofs, len);
@ -818,7 +829,7 @@ static int truncate_partial_data_page(struct inode *inode, u64 from,
if (IS_ERR(folio))
return PTR_ERR(folio) == -ENOENT ? 0 : PTR_ERR(folio);
truncate_out:
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
folio_zero_segment(folio, offset, folio_size(folio));
/* An encrypted inode should have a key and truncate the last page. */
@ -836,7 +847,7 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock)
struct f2fs_lock_context lc;
pgoff_t free_from;
int count = 0, err = 0;
struct folio *ifolio;
struct f2fs_cached_block *ientry;
bool truncate_page = false;
trace_f2fs_truncate_blocks_enter(inode, from);
@ -846,17 +857,17 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock)
goto out_err;
}
free_from = (pgoff_t)F2FS_BLK_ALIGN(from);
free_from = (pgoff_t)F2FS_BLK_ALIGN(sbi, from);
if (free_from >= max_file_blocks(inode))
if (free_from >= max_file_blocks(sbi, inode))
goto free_partial;
if (lock)
f2fs_lock_op(sbi, &lc);
ifolio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(ifolio)) {
err = PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(ientry)) {
err = PTR_ERR(ientry);
goto out;
}
@ -875,18 +886,18 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock)
f2fs_drop_extent_tree(inode);
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
goto out;
}
if (f2fs_has_inline_data(inode)) {
f2fs_truncate_inline_inode(inode, ifolio, from);
f2fs_folio_put(ifolio, true);
f2fs_truncate_inline_inode(inode, ientry, from);
f2fs_put_cache(ientry, true);
truncate_page = true;
goto out;
}
set_new_dnode(&dn, inode, ifolio, NULL, 0);
set_new_dnode(&dn, inode, ientry, NULL, 0);
err = f2fs_get_dnode_of_data(&dn, free_from, LOOKUP_NODE_RA);
if (err) {
if (err == -ENOENT)
@ -894,12 +905,12 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock)
goto out;
}
count = ADDRS_PER_PAGE(dn.node_folio, inode);
count = ADDRS_PER_PAGE(dn.node_entry, inode);
count -= dn.ofs_in_node;
f2fs_bug_on(sbi, count < 0);
if (dn.ofs_in_node || IS_INODE(dn.node_folio)) {
if (dn.ofs_in_node || IS_INODE(sbi, dn.node_entry)) {
f2fs_truncate_data_blocks_range(&dn, count);
free_from += count;
}
@ -959,9 +970,10 @@ int f2fs_truncate_blocks(struct inode *inode, u64 from, bool lock)
int f2fs_truncate(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
int err;
if (unlikely(f2fs_cp_error(F2FS_I_SB(inode))))
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
if (!(S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
@ -986,8 +998,8 @@ int f2fs_truncate(struct inode *inode)
* leak in evict() path.
*/
truncate_inode_pages_range(inode->i_mapping,
F2FS_BLK_TO_BYTES(0),
F2FS_BLK_END_BYTES(0));
F2FS_BLK_TO_BYTES(sbi, 0),
F2FS_BLK_END_BYTES(sbi, 0));
return err;
}
}
@ -1097,17 +1109,13 @@ static void __setattr_copy(const struct mnt_idmap *idmap,
struct inode *inode, const struct iattr *attr)
{
unsigned int ia_valid = attr->ia_valid;
struct iattr sattr = *attr;
i_uid_update(idmap, attr, inode);
i_gid_update(idmap, attr, inode);
if (ia_valid & ATTR_ATIME)
inode_set_atime_to_ts(inode, attr->ia_atime);
if (ia_valid & ATTR_MTIME)
inode_set_mtime_to_ts(inode, attr->ia_mtime);
if (ia_valid & ATTR_CTIME)
inode_set_ctime_to_ts(inode, attr->ia_ctime);
sattr.ia_valid &= ~ATTR_MODE;
setattr_copy(idmap, inode, &sattr);
if (ia_valid & ATTR_MODE) {
umode_t mode = attr->ia_mode;
umode_t mode = sattr.ia_mode;
if (!in_group_or_capable(idmap, inode, i_gid_into_vfsgid(idmap, inode)))
mode &= ~S_ISGID;
@ -1154,7 +1162,7 @@ int f2fs_setattr(const struct mnt_idmap *idmap, struct dentry *dentry,
return -EOPNOTSUPP;
if (is_inode_flag_set(inode, FI_COMPRESS_RELEASED) &&
!IS_ALIGNED(attr->ia_size,
F2FS_BLK_TO_BYTES(fi->i_cluster_size)))
F2FS_BLK_TO_BYTES(sbi, fi->i_cluster_size)))
return -EINVAL;
if (f2fs_is_pinned_file(inode)) {
@ -1170,7 +1178,7 @@ int f2fs_setattr(const struct mnt_idmap *idmap, struct dentry *dentry,
* pinned file.
*/
else if (!IS_ALIGNED(attr->ia_size,
F2FS_BLK_TO_BYTES(CAP_BLKS_PER_SEC(sbi))))
F2FS_BLK_TO_BYTES(sbi, CAP_BLKS_PER_SEC(sbi))))
return -EINVAL;
}
}
@ -1301,7 +1309,7 @@ static int fill_zero(struct inode *inode, pgoff_t index,
if (IS_ERR(folio))
return PTR_ERR(folio);
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
folio_zero_range(folio, start, len);
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
@ -1327,7 +1335,7 @@ int f2fs_truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end)
return err;
}
end_offset = ADDRS_PER_PAGE(dn.node_folio, inode);
end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
count = min(end_offset - dn.ofs_in_node, pg_end - pg_start);
f2fs_bug_on(F2FS_I_SB(inode), count == 0 || count > end_offset);
@ -1427,7 +1435,7 @@ static int __read_out_blkaddrs(struct inode *inode, block_t *blkaddr,
goto next;
}
done = min((pgoff_t)ADDRS_PER_PAGE(dn.node_folio, inode) -
done = min((pgoff_t)ADDRS_PER_PAGE(dn.node_entry, inode) -
dn.ofs_in_node, len);
for (i = 0; i < done; i++, blkaddr++, do_replace++, dn.ofs_in_node++) {
*blkaddr = f2fs_data_blkaddr(&dn);
@ -1516,7 +1524,7 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode,
}
ilen = min((pgoff_t)
ADDRS_PER_PAGE(dn.node_folio, dst_inode) -
ADDRS_PER_PAGE(dn.node_entry, dst_inode) -
dn.ofs_in_node, len - i);
do {
dn.data_blkaddr = f2fs_data_blkaddr(&dn);
@ -1554,7 +1562,7 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode,
return PTR_ERR(fdst);
}
f2fs_folio_wait_writeback(fdst, DATA, true, true);
f2fs_folio_wait_writeback(fdst, true, true);
memcpy_folio(fdst, 0, fsrc, 0, PAGE_SIZE);
folio_mark_dirty(fdst);
@ -1664,7 +1672,8 @@ static int f2fs_collapse_range(struct inode *inode, loff_t offset, loff_t len)
return -EINVAL;
/* collapse range should be aligned to block size of f2fs. */
if (offset & (F2FS_BLKSIZE - 1) || len & (F2FS_BLKSIZE - 1))
if (offset & F2FS_BLKSIZE_MASK(F2FS_I_SB(inode)) ||
len & F2FS_BLKSIZE_MASK(F2FS_I_SB(inode)))
return -EINVAL;
ret = f2fs_convert_inline_inode(inode);
@ -1823,7 +1832,7 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
goto out;
}
end_offset = ADDRS_PER_PAGE(dn.node_folio, inode);
end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
end = min(pg_end, end_offset - dn.ofs_in_node + index);
ret = f2fs_do_zero_range(&dn, index, end);
@ -1879,7 +1888,8 @@ static int f2fs_insert_range(struct inode *inode, loff_t offset, loff_t len)
return -EINVAL;
/* insert range should be aligned to block size of f2fs. */
if (offset & (F2FS_BLKSIZE - 1) || len & (F2FS_BLKSIZE - 1))
if (offset & F2FS_BLKSIZE_MASK(F2FS_I_SB(inode)) ||
len & F2FS_BLKSIZE_MASK(F2FS_I_SB(inode)))
return -EINVAL;
ret = f2fs_convert_inline_inode(inode);
@ -2107,9 +2117,11 @@ static long f2fs_fallocate(struct file *file, int mode,
/*
* Pinned file should not support partial truncation since the block
* can be used by applications.
* can be used by applications. Atomic files should not either, since
* these modify the original inode directly and break atomicity.
*/
if ((f2fs_compressed_file(inode) || f2fs_is_pinned_file(inode)) &&
if ((f2fs_compressed_file(inode) || f2fs_is_pinned_file(inode) ||
f2fs_is_atomic_file(inode)) &&
(mode & (FALLOC_FL_PUNCH_HOLE | FALLOC_FL_COLLAPSE_RANGE |
FALLOC_FL_ZERO_RANGE | FALLOC_FL_INSERT_RANGE))) {
ret = -EOPNOTSUPP;
@ -2397,6 +2409,8 @@ static int f2fs_ioc_start_atomic_write(struct file *filp, bool truncate)
f2fs_down_write(&fi->i_gc_rwsem[WRITE]);
f2fs_down_write(&fi->i_gc_rwsem[READ]);
/* serialize with f2fs_vm_page_mkwrite(), see the comment there */
filemap_invalidate_lock(inode->i_mapping);
/*
* Should wait end_io to count F2FS_WB_CP_DATA correctly by
@ -2445,13 +2459,14 @@ static int f2fs_ioc_start_atomic_write(struct file *filp, bool truncate)
fi->original_i_size = isize;
if (truncate) {
set_inode_flag(inode, FI_ATOMIC_REPLACE);
truncate_inode_pages_final(inode->i_mapping);
truncate_inode_pages(inode->i_mapping, 0);
f2fs_i_size_write(inode, 0);
isize = 0;
}
f2fs_i_size_write(fi->cow_inode, isize);
out_unlock:
filemap_invalidate_unlock(inode->i_mapping);
f2fs_up_write(&fi->i_gc_rwsem[READ]);
f2fs_up_write(&fi->i_gc_rwsem[WRITE]);
if (ret)
@ -2557,7 +2572,7 @@ int f2fs_do_shutdown(struct f2fs_sb_info *sbi, unsigned int flag,
f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN);
break;
case F2FS_GOING_DOWN_METAFLUSH:
f2fs_sync_meta_pages(sbi, LONG_MAX, FS_META_IO);
f2fs_sync_meta_caches(sbi, LONG_MAX, true, FS_META_IO);
f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_SHUTDOWN);
break;
case F2FS_GOING_DOWN_NEED_FSCK:
@ -2641,7 +2656,8 @@ static int f2fs_keep_noreuse_range(struct inode *inode,
loff_t offset, loff_t len)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
u64 max_bytes = F2FS_BLK_TO_BYTES(max_file_blocks(inode));
u64 max_bytes = F2FS_BLK_TO_BYTES(sbi,
max_file_blocks(sbi, inode));
u64 start, end;
int ret = 0;
@ -3029,8 +3045,8 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi,
/* writeback all dirty pages in the range */
err = filemap_write_and_wait_range(inode->i_mapping,
pg_start << PAGE_SHIFT,
(pg_end << PAGE_SHIFT) - 1);
(loff_t)pg_start << PAGE_SHIFT,
((loff_t)pg_end << PAGE_SHIFT) - 1);
if (err)
goto out;
@ -3122,7 +3138,7 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi,
goto clear_out;
}
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
folio_mark_dirty(folio);
folio_set_f2fs_gcing(folio);
@ -3181,11 +3197,12 @@ static int f2fs_ioc_defragment(struct file *filp, unsigned long arg)
return -EFAULT;
/* verify alignment of offset & size */
if (range.start & (F2FS_BLKSIZE - 1) || range.len & (F2FS_BLKSIZE - 1))
if (range.start & F2FS_BLKSIZE_MASK(sbi) ||
range.len & F2FS_BLKSIZE_MASK(sbi))
return -EINVAL;
if (unlikely((range.start + range.len) >> PAGE_SHIFT >
max_file_blocks(inode)))
if (unlikely(F2FS_BYTES_TO_BLK(sbi, range.start + range.len) >
max_file_blocks(sbi, inode)))
return -EINVAL;
err = mnt_want_write_file(filp);
@ -3267,7 +3284,7 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
if (src == dst && pos_out > pos_in && pos_out < pos_in + len)
goto out_unlock;
if (pos_in + len == src->i_size)
len = ALIGN(src->i_size, F2FS_BLKSIZE) - pos_in;
len = ALIGN(src->i_size, F2FS_BLKSIZE(sbi)) - pos_in;
if (len == 0) {
ret = 0;
goto out_unlock;
@ -3285,9 +3302,9 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
}
/* verify the end result is block aligned */
if (!IS_ALIGNED(pos_in, F2FS_BLKSIZE) ||
!IS_ALIGNED(pos_in + len, F2FS_BLKSIZE) ||
!IS_ALIGNED(pos_out, F2FS_BLKSIZE))
if (!IS_ALIGNED(pos_in, F2FS_BLKSIZE(sbi)) ||
!IS_ALIGNED(pos_in + len, F2FS_BLKSIZE(sbi)) ||
!IS_ALIGNED(pos_out, F2FS_BLKSIZE(sbi)))
goto out_unlock;
ret = f2fs_convert_inline_inode(src);
@ -3319,9 +3336,9 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
}
f2fs_lock_op(sbi, &lc);
ret = __exchange_data_block(src, dst, F2FS_BYTES_TO_BLK(pos_in),
F2FS_BYTES_TO_BLK(pos_out),
F2FS_BYTES_TO_BLK(len), false);
ret = __exchange_data_block(src, dst, F2FS_BYTES_TO_BLK(sbi, pos_in),
F2FS_BYTES_TO_BLK(sbi, pos_out),
F2FS_BYTES_TO_BLK(sbi, len), false);
if (!ret) {
if (dst_max_i_size)
@ -3840,10 +3857,11 @@ static int f2fs_ioc_reserve_dev_alias(struct file *filp)
write_unlock(&et->lock);
clear_inode_flag(inode, FI_NO_EXTENT);
f2fs_drop_discard_cmd_range(sbi, ei.blk, ei.len);
f2fs_reserve_device_alias(sbi, ei.blk, ei.len);
i_size_write(inode, (loff_t)ei.len << sbi->log_blocksize);
f2fs_update_inode_page(inode);
f2fs_update_inode_cache(inode);
spin_lock(&FREE_I(sbi)->segmap_lock);
FDEV(f2fs_target_device_index(sbi, ei.blk)).is_reserving = false;
@ -3937,7 +3955,7 @@ static int f2fs_ioc_release_dev_alias(struct file *filp)
goto out_inode_unlock;
}
f2fs_update_inode_page(inode);
f2fs_update_inode_cache(inode);
f2fs_unlock_op(sbi, &lc);
f2fs_up_write_trace(&sbi->gc_lock, &glc);
@ -3987,7 +4005,7 @@ int f2fs_precache_extents(struct inode *inode)
map.m_next_extent = &m_next_extent;
map.m_seg_type = NO_CHECK_TYPE;
map.m_may_create = false;
end = F2FS_BLK_ALIGN(i_size_read(inode));
end = F2FS_BLK_ALIGN(F2FS_I_SB(inode), i_size_read(inode));
while (map.m_lblk < end) {
map.m_len = end - map.m_lblk;
@ -4156,7 +4174,7 @@ static int release_compress_blocks(struct dnode_of_data *dn, pgoff_t count)
int i;
for (i = 0; i < count; i++) {
blkaddr = data_blkaddr(dn->inode, dn->node_folio,
blkaddr = data_blkaddr(dn->inode, dn->node_entry,
dn->ofs_in_node + i);
if (!__is_valid_data_blkaddr(blkaddr))
@ -4275,7 +4293,7 @@ static int f2fs_release_compress_blocks(struct file *filp, unsigned long arg)
break;
}
end_offset = ADDRS_PER_PAGE(dn.node_folio, inode);
end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
count = min(end_offset - dn.ofs_in_node, last_idx - page_idx);
count = round_up(count, fi->i_cluster_size);
@ -4326,7 +4344,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count,
int i;
for (i = 0; i < count; i++) {
blkaddr = data_blkaddr(dn->inode, dn->node_folio,
blkaddr = data_blkaddr(dn->inode, dn->node_entry,
dn->ofs_in_node + i);
if (!__is_valid_data_blkaddr(blkaddr))
@ -4343,7 +4361,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count,
int ret;
for (i = 0; i < cluster_size; i++) {
blkaddr = data_blkaddr(dn->inode, dn->node_folio,
blkaddr = data_blkaddr(dn->inode, dn->node_entry,
dn->ofs_in_node + i);
if (i == 0) {
@ -4454,7 +4472,7 @@ static int f2fs_reserve_compress_blocks(struct file *filp, unsigned long arg)
break;
}
end_offset = ADDRS_PER_PAGE(dn.node_folio, inode);
end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
count = min(end_offset - dn.ofs_in_node, last_idx - page_idx);
count = round_up(count, fi->i_cluster_size);
@ -4503,8 +4521,8 @@ static int f2fs_reserve_compress_blocks(struct file *filp, unsigned long arg)
static int f2fs_secure_erase(struct block_device *bdev, struct inode *inode,
pgoff_t off, block_t block, block_t len, u32 flags)
{
sector_t sector = SECTOR_FROM_BLOCK(block);
sector_t nr_sects = SECTOR_FROM_BLOCK(len);
sector_t sector = SECTOR_FROM_BLOCK(F2FS_I_SB(inode), block);
sector_t nr_sects = SECTOR_FROM_BLOCK(F2FS_I_SB(inode), len);
int ret = 0;
if (flags & F2FS_TRIM_FILE_DISCARD) {
@ -4581,14 +4599,14 @@ static int f2fs_sec_trim_file(struct file *filp, unsigned long arg)
to_end = true;
}
if (!IS_ALIGNED(range.start, F2FS_BLKSIZE) ||
(!to_end && !IS_ALIGNED(end_addr, F2FS_BLKSIZE))) {
if (!IS_ALIGNED(range.start, F2FS_BLKSIZE(sbi)) ||
(!to_end && !IS_ALIGNED(end_addr, F2FS_BLKSIZE(sbi)))) {
ret = -EINVAL;
goto err;
}
index = F2FS_BYTES_TO_BLK(range.start);
pg_end = DIV_ROUND_UP(end_addr, F2FS_BLKSIZE);
index = F2FS_BYTES_TO_BLK(sbi, range.start);
pg_end = F2FS_BLK_ALIGN(sbi, end_addr);
ret = f2fs_convert_inline_inode(inode);
if (ret)
@ -4620,7 +4638,7 @@ static int f2fs_sec_trim_file(struct file *filp, unsigned long arg)
goto out;
}
end_offset = ADDRS_PER_PAGE(dn.node_folio, inode);
end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
count = min(end_offset - dn.ofs_in_node, pg_end - index);
for (i = 0; i < count; i++, index++, dn.ofs_in_node++) {
struct block_device *cur_bdev;
@ -4816,7 +4834,7 @@ static int redirty_blocks(struct inode *inode, pgoff_t page_idx, int len)
/* It will never fail, when folio has pinned above */
f2fs_bug_on(F2FS_I_SB(inode), IS_ERR(folio));
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
folio_mark_dirty(folio);
folio_set_f2fs_gcing(folio);
@ -5168,7 +5186,7 @@ static int f2fs_dio_read_end_io(struct kiocb *iocb, ssize_t size, int error,
{
struct f2fs_sb_info *sbi = F2FS_I_SB(file_inode(iocb->ki_filp));
dec_page_count(sbi, F2FS_DIO_READ);
dec_cache_count(sbi, F2FS_DIO_READ);
if (error)
return error;
f2fs_update_iostat(sbi, NULL, APP_DIRECT_READ_IO, size);
@ -5226,13 +5244,13 @@ static ssize_t f2fs_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
* the higher-level function iomap_dio_rw() in order to ensure that the
* F2FS_DIO_READ counter will be decremented correctly in all cases.
*/
inc_page_count(sbi, F2FS_DIO_READ);
inc_cache_count(sbi, F2FS_DIO_READ);
dio = __iomap_dio_rw(iocb, to, &f2fs_iomap_ops,
&f2fs_iomap_dio_read_ops, 0, NULL, 0);
if (IS_ERR_OR_NULL(dio)) {
ret = PTR_ERR_OR_ZERO(dio);
if (ret != -EIOCBQUEUED)
dec_page_count(sbi, F2FS_DIO_READ);
dec_cache_count(sbi, F2FS_DIO_READ);
} else {
ret = iomap_dio_complete(dio);
}
@ -5285,7 +5303,7 @@ static ssize_t f2fs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
/* In LFS mode, if there is inflight dio, wait for its completion */
if (f2fs_lfs_mode(F2FS_I_SB(inode)) &&
get_pages(F2FS_I_SB(inode), F2FS_DIO_WRITE) &&
get_nr_caches(F2FS_I_SB(inode), F2FS_DIO_WRITE) &&
(!f2fs_is_pinned_file(inode) || !dio))
inode_dio_wait(inode);
@ -5378,7 +5396,8 @@ static int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *iter,
* buffered IO, if DIO meets any holes.
*/
if (dio && i_size_read(inode) &&
(F2FS_BYTES_TO_BLK(pos) < F2FS_BLK_ALIGN(i_size_read(inode))))
(F2FS_BYTES_TO_BLK(sbi, pos) <
F2FS_BLK_ALIGN(sbi, i_size_read(inode))))
return 0;
/* No-wait I/O can't allocate blocks. */
@ -5399,8 +5418,8 @@ static int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *iter,
}
/* Do not preallocate blocks that will be written partially in 4KB. */
map.m_lblk = F2FS_BLK_ALIGN(pos);
map.m_len = F2FS_BYTES_TO_BLK(pos + count);
map.m_lblk = F2FS_BLK_ALIGN(sbi, pos);
map.m_len = F2FS_BYTES_TO_BLK(sbi, pos + count);
if (map.m_len > map.m_lblk)
map.m_len -= map.m_lblk;
else
@ -5450,7 +5469,7 @@ static int f2fs_dio_write_end_io(struct kiocb *iocb, ssize_t size, int error,
{
struct f2fs_sb_info *sbi = F2FS_I_SB(file_inode(iocb->ki_filp));
dec_page_count(sbi, F2FS_DIO_WRITE);
dec_cache_count(sbi, F2FS_DIO_WRITE);
if (error)
return error;
f2fs_update_time(sbi, REQ_TIME);
@ -5561,7 +5580,7 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
* the higher-level function iomap_dio_rw() in order to ensure that the
* F2FS_DIO_WRITE counter will be decremented correctly in all cases.
*/
inc_page_count(sbi, F2FS_DIO_WRITE);
inc_cache_count(sbi, F2FS_DIO_WRITE);
dio_flags = 0;
if (pos + count > inode->i_size)
dio_flags |= IOMAP_DIO_FORCE_WAIT;
@ -5572,7 +5591,7 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
if (ret == -ENOTBLK)
ret = 0;
if (ret != -EIOCBQUEUED)
dec_page_count(sbi, F2FS_DIO_WRITE);
dec_cache_count(sbi, F2FS_DIO_WRITE);
} else {
ret = iomap_dio_complete(dio);
}

242
fs/f2fs/gc.c
View File

@ -1054,7 +1054,7 @@ static int gc_node_segment(struct f2fs_sb_info *sbi,
for (off = 0; off < usable_blks_in_seg; off++, entry++) {
nid_t nid = le32_to_cpu(entry->nid);
struct folio *node_folio;
struct f2fs_cached_block *node_entry;
struct node_info ni;
int err;
@ -1066,38 +1066,38 @@ static int gc_node_segment(struct f2fs_sb_info *sbi,
continue;
if (phase == 0) {
f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1,
f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(sbi, nid), 1,
META_NAT, true);
continue;
}
if (phase == 1) {
f2fs_ra_node_page(sbi, nid);
f2fs_ra_node_cache(sbi, nid);
continue;
}
/* phase == 2 */
node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR);
if (IS_ERR(node_folio))
node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR);
if (IS_ERR(node_entry))
continue;
/* block may become invalid during f2fs_get_node_folio */
if (check_valid_map(sbi, segno, off) == 0) {
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
continue;
}
if (f2fs_get_node_info(sbi, nid, &ni, false)) {
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
continue;
}
if (ni.blk_addr != start_addr + off) {
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
continue;
}
err = f2fs_move_node_folio(node_folio, gc_type);
err = f2fs_move_node_cache(node_entry, gc_type);
if (!err && gc_type == FG_GC)
submitted++;
stat_inc_node_blk_count(sbi, 1, gc_type);
@ -1123,7 +1123,8 @@ static int gc_node_segment(struct f2fs_sb_info *sbi,
*/
block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode)
{
unsigned int indirect_blks = 2 * NIDS_PER_BLOCK + 4;
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
unsigned int indirect_blks = 2 * NIDS_PER_BLOCK(sbi) + 4;
unsigned int bidx;
if (node_ofs == 0)
@ -1132,11 +1133,12 @@ block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode)
if (node_ofs <= 2) {
bidx = node_ofs - 1;
} else if (node_ofs <= indirect_blks) {
int dec = (node_ofs - 4) / (NIDS_PER_BLOCK + 1);
int dec = (node_ofs - 4) / (NIDS_PER_BLOCK(sbi) + 1);
bidx = node_ofs - 2 - dec;
} else {
int dec = (node_ofs - indirect_blks - 3) / (NIDS_PER_BLOCK + 1);
int dec = (node_ofs - indirect_blks - 3) /
(NIDS_PER_BLOCK(sbi) + 1);
bidx = node_ofs - 5 - dec;
}
@ -1146,7 +1148,7 @@ block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode)
static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
struct node_info *dni, block_t blkaddr, unsigned int *nofs)
{
struct folio *node_folio;
struct f2fs_cached_block *node_entry;
nid_t nid;
unsigned int ofs_in_node, max_addrs, base;
block_t source_blkaddr;
@ -1154,12 +1156,12 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
nid = le32_to_cpu(sum->nid);
ofs_in_node = le16_to_cpu(sum->ofs_in_node);
node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR);
if (IS_ERR(node_folio))
node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR);
if (IS_ERR(node_entry))
return false;
if (f2fs_get_node_info(sbi, nid, dni, false)) {
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
return false;
}
@ -1170,28 +1172,28 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
}
if (f2fs_check_nid_range(sbi, dni->ino)) {
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
return false;
}
if (IS_INODE(node_folio)) {
base = offset_in_addr(F2FS_INODE(node_folio));
max_addrs = DEF_ADDRS_PER_INODE;
if (IS_INODE(sbi, node_entry)) {
base = offset_in_addr(F2FS_INODE(node_entry));
max_addrs = DEF_ADDRS_PER_INODE(sbi);
} else {
base = 0;
max_addrs = DEF_ADDRS_PER_BLOCK;
max_addrs = DEF_ADDRS_PER_BLOCK(sbi);
}
if (base + ofs_in_node >= max_addrs) {
f2fs_err(sbi, "Inconsistent blkaddr offset: base:%u, ofs_in_node:%u, max:%u, ino:%u, nid:%u",
base, ofs_in_node, max_addrs, dni->ino, dni->nid);
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
return false;
}
*nofs = ofs_of_node(node_folio);
source_blkaddr = data_blkaddr(NULL, node_folio, ofs_in_node);
f2fs_folio_put(node_folio, true);
*nofs = ofs_of_node(sbi, node_entry);
source_blkaddr = data_blkaddr(NULL, node_entry, ofs_in_node);
f2fs_put_cache(node_entry, true);
if (source_blkaddr != blkaddr) {
#ifdef CONFIG_F2FS_CHECK_FS
@ -1217,7 +1219,8 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
struct address_space *mapping = inode->i_mapping;
struct inode *atomic_inode = NULL;
struct dnode_of_data dn;
struct folio *folio, *efolio;
struct folio *folio;
struct f2fs_cached_block *entry;
struct f2fs_io_info fio = {
.sbi = sbi,
.ino = inode->i_ino,
@ -1227,6 +1230,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
.op_flags = 0,
.encrypted_page = NULL,
.in_list = 0,
.is_cache = 1,
};
int err = 0;
@ -1281,36 +1285,35 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
* don't cache encrypted data into meta inode until previous dirty
* data were writebacked to avoid racing between GC and flush.
*/
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
f2fs_wait_on_block_writeback(inode, dn.data_blkaddr);
efolio = f2fs_filemap_get_folio(META_MAPPING(sbi), dn.data_blkaddr,
FGP_LOCK | FGP_CREAT, GFP_NOFS);
if (IS_ERR(efolio)) {
err = PTR_ERR(efolio);
entry = f2fs_grab_cache(META_CACHE(sbi), dn.data_blkaddr,
F2FS_CACHE_LOCK_CREATE);
if (IS_ERR(entry)) {
err = PTR_ERR(entry);
goto put_folio;
}
fio.encrypted_page = &efolio->page;
if (f2fs_cache_test_uptodate(entry))
goto put_cache;
if (folio_test_uptodate(efolio))
goto put_encrypted_page;
err = f2fs_submit_page_bio(&fio);
fio.cache_entry = entry;
err = f2fs_submit_cache_read(&fio);
if (err)
goto put_encrypted_page;
f2fs_put_page(fio.encrypted_page, false);
goto put_cache;
f2fs_put_cache(entry, false);
f2fs_folio_put(folio, true);
f2fs_update_iostat(sbi, inode, FS_DATA_READ_IO, F2FS_BLKSIZE);
f2fs_update_iostat(sbi, NULL, FS_GDATA_READ_IO, F2FS_BLKSIZE);
f2fs_update_iostat(sbi, inode, FS_DATA_READ_IO, F2FS_BLKSIZE(sbi));
f2fs_update_iostat(sbi, NULL, FS_GDATA_READ_IO, F2FS_BLKSIZE(sbi));
if (atomic_inode)
iput(atomic_inode);
return 0;
put_encrypted_page:
f2fs_put_page(fio.encrypted_page, true);
put_cache:
f2fs_put_cache(entry, true);
put_folio:
f2fs_folio_put(folio, true);
out_iput:
@ -1320,7 +1323,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
}
/*
* Move data block via META_MAPPING while keeping locked data page.
* Move data block via meta cache while keeping locked data page.
* This can be used to move blocks, aka LBAs, directly on disk.
*/
static int move_data_block(struct inode *inode, block_t bidx,
@ -1337,11 +1340,13 @@ static int move_data_block(struct inode *inode, block_t bidx,
.op_flags = 0,
.encrypted_page = NULL,
.in_list = 0,
.is_cache = 1,
};
struct dnode_of_data dn;
struct f2fs_summary sum;
struct node_info ni;
struct folio *folio, *mfolio, *efolio;
struct folio *folio;
struct f2fs_cached_block *sentry, *tentry;
block_t newaddr;
int err = 0;
bool lfs_mode = f2fs_lfs_mode(fio.sbi);
@ -1391,7 +1396,7 @@ static int move_data_block(struct inode *inode, block_t bidx,
* don't cache encrypted data into meta inode until previous dirty
* data were writebacked to avoid racing between GC and flush.
*/
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
f2fs_wait_on_block_writeback(inode, dn.data_blkaddr);
@ -1406,33 +1411,33 @@ static int move_data_block(struct inode *inode, block_t bidx,
if (lfs_mode)
f2fs_down_write(&fio.sbi->io_order_lock);
mfolio = f2fs_grab_cache_folio(META_MAPPING(fio.sbi),
fio.old_blkaddr, false);
if (IS_ERR(mfolio)) {
err = PTR_ERR(mfolio);
sentry = f2fs_grab_cache(META_CACHE(fio.sbi), fio.old_blkaddr,
F2FS_CACHE_LOCK_CREATE);
if (IS_ERR(sentry)) {
err = PTR_ERR(sentry);
goto up_out;
}
fio.encrypted_page = folio_file_page(mfolio, fio.old_blkaddr);
fio.cache_entry = sentry;
/* read source block in mfolio */
if (!folio_test_uptodate(mfolio)) {
err = f2fs_submit_page_bio(&fio);
/* read source block in cache */
if (!f2fs_cache_test_uptodate(sentry)) {
err = f2fs_submit_cache_read(&fio);
if (err) {
f2fs_folio_put(mfolio, true);
f2fs_put_cache(sentry, true);
goto up_out;
}
f2fs_update_iostat(fio.sbi, inode, FS_DATA_READ_IO,
F2FS_BLKSIZE);
F2FS_BLKSIZE(fio.sbi));
f2fs_update_iostat(fio.sbi, NULL, FS_GDATA_READ_IO,
F2FS_BLKSIZE);
F2FS_BLKSIZE(fio.sbi));
folio_lock(mfolio);
if (unlikely(!is_meta_folio(mfolio) ||
!folio_test_uptodate(mfolio))) {
f2fs_lock_cache(sentry);
if (unlikely(!f2fs_is_meta_cache(sentry) ||
!f2fs_cache_test_uptodate(sentry))) {
err = -EIO;
f2fs_folio_put(mfolio, true);
f2fs_put_cache(sentry, true);
goto up_out;
}
}
@ -1440,49 +1445,58 @@ static int move_data_block(struct inode *inode, block_t bidx,
set_summary(&sum, dn.nid, dn.ofs_in_node, ni.version);
/* allocate block address */
err = f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr,
err = f2fs_allocate_data_block(fio.sbi, fio.old_blkaddr, &newaddr,
&sum, type, NULL);
if (err) {
f2fs_folio_put(mfolio, true);
f2fs_put_cache(sentry, true);
/* filesystem should shutdown, no need to recovery block */
goto up_out;
}
efolio = f2fs_filemap_get_folio(META_MAPPING(fio.sbi), newaddr,
FGP_LOCK | FGP_CREAT, GFP_NOFS);
if (IS_ERR(efolio)) {
err = PTR_ERR(efolio);
f2fs_folio_put(mfolio, true);
/*
* If newaddr is in a sequential zone, it must be written once it is
* allocated, otherwise it leaves a hole in the zone, and the following
* write to the zone will fail as an unaligned write, so don't fail
* here.
*/
tentry = f2fs_grab_cache(META_CACHE(fio.sbi), newaddr,
F2FS_CACHE_LOCK_CREATE |
(f2fs_is_sequential_zone_area(fio.sbi, newaddr) ?
F2FS_CACHE_NOFAIL : 0));
if (IS_ERR(tentry)) {
err = PTR_ERR(tentry);
f2fs_put_cache(sentry, true);
goto recover_block;
}
fio.encrypted_page = &efolio->page;
fio.cache_entry = tentry;
/* write target block */
f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true, true);
memcpy(page_address(fio.encrypted_page),
folio_address(mfolio), PAGE_SIZE);
f2fs_folio_put(mfolio, true);
f2fs_cache_wait_writeback_cond(tentry, DATA);
memcpy(cache_address(tentry), cache_address(sentry),
fio.sbi->blocksize);
f2fs_put_cache(sentry, true);
f2fs_invalidate_internal_cache(fio.sbi, fio.old_blkaddr, 1);
set_page_dirty(fio.encrypted_page);
if (clear_page_dirty_for_io(fio.encrypted_page))
dec_page_count(fio.sbi, F2FS_DIRTY_META);
f2fs_mark_cache_dirty(tentry);
if (f2fs_cache_test_and_clear_dirty(tentry))
dec_cache_count(fio.sbi, F2FS_DIRTY_META);
set_page_writeback(fio.encrypted_page);
f2fs_start_cache_writeback(tentry);
fio.op = REQ_OP_WRITE;
fio.op_flags = REQ_SYNC;
fio.new_blkaddr = newaddr;
f2fs_submit_page_write(&fio);
f2fs_submit_cache_write(&fio);
f2fs_update_iostat(fio.sbi, NULL, FS_GC_DATA_IO, F2FS_BLKSIZE);
f2fs_update_iostat(fio.sbi, NULL, FS_GC_DATA_IO,
F2FS_BLKSIZE(fio.sbi));
f2fs_update_data_blkaddr(&dn, newaddr);
set_inode_flag(inode, FI_APPEND_WRITE);
f2fs_put_page(fio.encrypted_page, true);
f2fs_put_cache(tentry, true);
recover_block:
if (err)
f2fs_do_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr,
@ -1543,7 +1557,7 @@ static int move_data_page(struct inode *inode, block_t bidx, int gc_type,
bool is_dirty = folio_test_dirty(folio);
retry:
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
folio_mark_dirty(folio);
if (folio_clear_dirty_for_io(folio)) {
@ -1614,13 +1628,13 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
continue;
if (phase == 0) {
f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1,
f2fs_ra_meta_caches(sbi, NAT_BLOCK_OFFSET(sbi, nid), 1,
META_NAT, true);
continue;
}
if (phase == 1) {
f2fs_ra_node_page(sbi, nid);
f2fs_ra_node_cache(sbi, nid);
continue;
}
@ -1629,7 +1643,7 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
continue;
if (phase == 2) {
f2fs_ra_node_page(sbi, dni.ino);
f2fs_ra_node_cache(sbi, dni.ino);
continue;
}
@ -1818,28 +1832,31 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
sum_blk_cnt = DIV_ROUND_UP(end_segno - segno, sbi->sums_per_block);
/* readahead multi ssa blocks those have contiguous address */
if (__is_large_section(sbi))
f2fs_ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno),
f2fs_ra_meta_caches(sbi, GET_SUM_BLOCK(sbi, segno),
sum_blk_cnt, META_SSA, true);
/* reference all summary page */
while (segno < end_segno) {
struct folio *sum_folio = f2fs_get_sum_folio(sbi, segno);
struct f2fs_cached_block *sum_entry =
f2fs_get_sum_cache(sbi, segno);
segno += sbi->sums_per_block;
if (IS_ERR(sum_folio)) {
int err = PTR_ERR(sum_folio);
if (IS_ERR(sum_entry)) {
int err = PTR_ERR(sum_entry);
end_segno = segno - sbi->sums_per_block;
segno = rounddown(start_segno, sbi->sums_per_block);
while (segno < end_segno) {
sum_folio = filemap_get_folio(META_MAPPING(sbi),
sum_entry = f2fs_find_meta_cache(sbi,
GET_SUM_BLOCK(sbi, segno));
folio_put_refs(sum_folio, 2);
f2fs_put_cache(sum_entry, false);
f2fs_put_cache(sum_entry, false);
segno += sbi->sums_per_block;
}
return err;
}
folio_unlock(sum_folio);
f2fs_unlock_cache(sum_entry);
}
blk_start_plug(&plug);
@ -1847,16 +1864,24 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
segno = start_segno;
while (segno < end_segno) {
unsigned int cur_segno;
unsigned int block_end_segno;
/* find segment summary of victim */
struct folio *sum_folio = filemap_get_folio(META_MAPPING(sbi),
struct f2fs_cached_block *sum_entry =
f2fs_find_meta_cache(sbi,
GET_SUM_BLOCK(sbi, segno));
unsigned int block_end_segno = rounddown(segno, sbi->sums_per_block)
f2fs_bug_on(sbi, IS_ERR(sum_entry));
block_end_segno = rounddown(segno, sbi->sums_per_block)
+ sbi->sums_per_block;
if (block_end_segno > end_segno)
block_end_segno = end_segno;
if (unlikely(freezing(current)))
goto next_block;
if (is_cursec(sbi, GET_SEC_FROM_SEG(sbi, segno))) {
f2fs_err(sbi, "%s: segment %u is used by log",
__func__, segno);
@ -1864,8 +1889,8 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
goto next_block;
}
if (!folio_test_uptodate(sum_folio) ||
unlikely(f2fs_cp_error(sbi)))
if (!f2fs_cache_test_uptodate(sum_entry) ||
unlikely(f2fs_cp_error(sbi)))
goto next_block;
for (cur_segno = segno; cur_segno < block_end_segno;
@ -1884,7 +1909,7 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
data_type = (type == SUM_TYPE_DATA) ? DATA : NODE;
}
sum = SUM_BLK_PAGE_ADDR(sbi, sum_folio, cur_segno);
sum = SUM_BLK_ENTRY_ADDR(sbi, sum_entry, cur_segno);
if (type != GET_SUM_TYPE(sum_footer(sbi, sum))) {
f2fs_err(sbi, "Inconsistent segment (%u) type "
"[%d, %d] in SIT and SSA",
@ -1924,18 +1949,13 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
sbi->next_victim_seg[gc_type] =
(cur_segno + 1 < sec_end_segno) ?
cur_segno + 1 : NULL_SEGNO;
if (unlikely(freezing(current))) {
folio_put_refs(sum_folio, 2);
goto stop;
}
}
next_block:
folio_put_refs(sum_folio, 2);
f2fs_put_cache(sum_entry, false);
f2fs_put_cache(sum_entry, false);
segno = block_end_segno;
}
stop:
if (submitted)
f2fs_submit_merged_write(sbi, data_type);
@ -1963,9 +1983,9 @@ int f2fs_gc(struct f2fs_sb_info *sbi, struct f2fs_gc_control *gc_control)
trace_f2fs_gc_begin(sbi->sb, gc_type, gc_control->no_bg_gc,
gc_control->nr_free_secs,
get_pages(sbi, F2FS_DIRTY_NODES),
get_pages(sbi, F2FS_DIRTY_DENTS),
get_pages(sbi, F2FS_DIRTY_IMETA),
get_nr_caches(sbi, F2FS_DIRTY_NODES),
get_nr_caches(sbi, F2FS_DIRTY_DENTS),
get_nr_caches(sbi, F2FS_DIRTY_IMETA),
free_sections(sbi),
free_segments(sbi),
reserved_segments(sbi),
@ -2089,9 +2109,9 @@ int f2fs_gc(struct f2fs_sb_info *sbi, struct f2fs_gc_control *gc_control)
f2fs_unpin_all_sections(sbi, true);
trace_f2fs_gc_end(sbi->sb, ret, total_freed, total_sec_freed,
get_pages(sbi, F2FS_DIRTY_NODES),
get_pages(sbi, F2FS_DIRTY_DENTS),
get_pages(sbi, F2FS_DIRTY_IMETA),
get_nr_caches(sbi, F2FS_DIRTY_NODES),
get_nr_caches(sbi, F2FS_DIRTY_DENTS),
get_nr_caches(sbi, F2FS_DIRTY_IMETA),
free_sections(sbi),
free_segments(sbi),
reserved_segments(sbi),
@ -2326,6 +2346,7 @@ static void update_fs_metadata(struct f2fs_sb_info *sbi, int secs)
SM_I(sbi)->segment_count = (int)SM_I(sbi)->segment_count + segs;
MAIN_SEGS(sbi) = (int)MAIN_SEGS(sbi) + segs;
MAIN_SECS(sbi) += secs;
f2fs_adjust_pinned_area_boundary(sbi);
if (sbi->allocate_section_hint > MAIN_SECS(sbi))
sbi->allocate_section_hint = MAIN_SECS(sbi);
FREE_I(sbi)->free_sections = (int)FREE_I(sbi)->free_sections + secs;
@ -2357,7 +2378,7 @@ int f2fs_resize_fs(struct file *filp, __u64 block_count)
struct f2fs_lock_context lc;
struct f2fs_lock_context glc;
struct f2fs_lock_context clc;
unsigned int secs;
unsigned int secs, old_tail;
int err = 0;
__u32 rem;
@ -2466,6 +2487,8 @@ int f2fs_resize_fs(struct file *filp, __u64 block_count)
goto recover_out;
}
old_tail = F2FS_OPTION(sbi).resizable_tail_secno;
F2FS_OPTION(sbi).resizable_tail_secno -= min(old_tail, secs);
update_fs_metadata(sbi, -secs);
clear_sbi_flag(sbi, SBI_IS_RESIZEFS);
set_sbi_flag(sbi, SBI_IS_DIRTY);
@ -2473,6 +2496,7 @@ int f2fs_resize_fs(struct file *filp, __u64 block_count)
stat_inc_cp_call_count(sbi, TOTAL_CALL);
err = f2fs_write_checkpoint(sbi, &cpc);
if (err) {
F2FS_OPTION(sbi).resizable_tail_secno = old_tail;
update_fs_metadata(sbi, secs);
update_sb_metadata(sbi, secs);
f2fs_commit_super(sbi, false);

293
fs/f2fs/inline.c
View File

@ -34,27 +34,26 @@ bool f2fs_may_inline_data(struct inode *inode)
return !f2fs_post_read_required(inode);
}
static bool inode_has_blocks(struct inode *inode, struct folio *ifolio)
static bool inode_has_blocks(struct inode *inode, struct f2fs_cached_block *ientry)
{
struct f2fs_inode *ri = F2FS_INODE(ifolio);
int i;
if (F2FS_HAS_BLOCKS(inode))
return true;
for (i = 0; i < DEF_NIDS_PER_INODE; i++) {
if (ri->i_nid[i])
if (F2FS_INODE_NIDS(F2FS_I_SB(inode), ientry)[i])
return true;
}
return false;
}
bool f2fs_sanity_check_inline_data(struct inode *inode, struct folio *ifolio)
bool f2fs_sanity_check_inline_data(struct inode *inode, struct f2fs_cached_block *ientry)
{
if (!f2fs_has_inline_data(inode))
return false;
if (inode_has_blocks(inode, ifolio))
if (inode_has_blocks(inode, ientry))
return false;
if (!support_inline_data(inode))
@ -80,7 +79,7 @@ bool f2fs_may_inline_dentry(struct inode *inode)
return true;
}
void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio)
void f2fs_do_read_inline_data(struct folio *folio, struct f2fs_cached_block *ientry)
{
struct inode *inode = folio->mapping->host;
@ -92,13 +91,13 @@ void f2fs_do_read_inline_data(struct folio *folio, struct folio *ifolio)
folio_zero_segment(folio, MAX_INLINE_DATA(inode), folio_size(folio));
/* Copy the whole inline data block */
memcpy_to_folio(folio, 0, inline_data_addr(inode, ifolio),
memcpy_to_folio(folio, 0, inline_data_addr(inode, ientry),
MAX_INLINE_DATA(inode));
if (!folio_test_uptodate(folio))
folio_mark_uptodate(folio);
}
void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio,
void f2fs_truncate_inline_inode(struct inode *inode, struct f2fs_cached_block *ientry,
u64 from)
{
void *addr;
@ -106,11 +105,11 @@ void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio,
if (from >= MAX_INLINE_DATA(inode))
return;
addr = inline_data_addr(inode, ifolio);
addr = inline_data_addr(inode, ientry);
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_cache_wait_writeback(ientry);
memset(addr + from, 0, MAX_INLINE_DATA(inode) - from);
folio_mark_dirty(ifolio);
f2fs_mark_cache_dirty(ientry);
if (from == 0)
clear_inode_flag(inode, FI_DATA_EXIST);
@ -118,27 +117,27 @@ void f2fs_truncate_inline_inode(struct inode *inode, struct folio *ifolio,
int f2fs_read_inline_data(struct inode *inode, struct folio *folio)
{
struct folio *ifolio;
struct f2fs_cached_block *ientry;
ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ifolio)) {
ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ientry)) {
folio_unlock(folio);
return PTR_ERR(ifolio);
return PTR_ERR(ientry);
}
if (!f2fs_has_inline_data(inode)) {
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
return -EAGAIN;
}
if (folio->index)
folio_zero_segment(folio, 0, folio_size(folio));
else
f2fs_do_read_inline_data(folio, ifolio);
f2fs_do_read_inline_data(folio, ientry);
if (!folio_test_uptodate(folio))
folio_mark_uptodate(folio);
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
folio_unlock(folio);
return 0;
}
@ -186,7 +185,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio)
f2fs_bug_on(F2FS_F_SB(folio), folio_test_writeback(folio));
f2fs_do_read_inline_data(folio, dn->inode_folio);
f2fs_do_read_inline_data(folio, dn->inode_entry);
folio_mark_dirty(folio);
/* clear dirty state */
@ -197,7 +196,7 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio)
fio.old_blkaddr = dn->data_blkaddr;
set_inode_flag(dn->inode, FI_HOT_DATA);
f2fs_outplace_write_data(dn, &fio);
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
if (dirty) {
inode_dec_dirty_pages(dn->inode);
f2fs_remove_dirty_inode(dn->inode);
@ -207,8 +206,8 @@ int f2fs_convert_inline_folio(struct dnode_of_data *dn, struct folio *folio)
set_inode_flag(dn->inode, FI_APPEND_WRITE);
/* clear inline data and flag after data writeback */
f2fs_truncate_inline_inode(dn->inode, dn->inode_folio, 0);
folio_clear_f2fs_inline(dn->inode_folio);
f2fs_truncate_inline_inode(dn->inode, dn->inode_entry, 0);
f2fs_cache_clear_inline(dn->inode_entry);
clear_out:
stat_dec_inline_inode(dn->inode);
clear_inode_flag(dn->inode, FI_INLINE_DATA);
@ -221,7 +220,8 @@ int f2fs_convert_inline_inode(struct inode *inode)
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct dnode_of_data dn;
struct f2fs_lock_context lc;
struct folio *ifolio, *folio;
struct f2fs_cached_block *ientry;
struct folio *folio;
int err = 0;
if (f2fs_hw_is_readonly(sbi) || f2fs_readonly(sbi->sb))
@ -240,13 +240,13 @@ int f2fs_convert_inline_inode(struct inode *inode)
f2fs_lock_op(sbi, &lc);
ifolio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(ifolio)) {
err = PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(ientry)) {
err = PTR_ERR(ientry);
goto out;
}
set_new_dnode(&dn, inode, ifolio, ifolio, 0);
set_new_dnode(&dn, inode, ientry, ientry, 0);
if (f2fs_has_inline_data(inode))
err = f2fs_convert_inline_folio(&dn, folio);
@ -266,35 +266,35 @@ int f2fs_convert_inline_inode(struct inode *inode)
int f2fs_write_inline_data(struct inode *inode, struct folio *folio)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct folio *ifolio;
struct f2fs_cached_block *ientry;
ifolio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
if (!f2fs_has_inline_data(inode)) {
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
return -EAGAIN;
}
f2fs_bug_on(F2FS_I_SB(inode), folio->index);
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
memcpy_from_folio(inline_data_addr(inode, ifolio),
f2fs_cache_wait_writeback(ientry);
memcpy_from_folio(inline_data_addr(inode, ientry),
folio, 0, MAX_INLINE_DATA(inode));
folio_mark_dirty(ifolio);
f2fs_mark_cache_dirty(ientry);
f2fs_clear_page_cache_dirty_tag(folio);
set_inode_flag(inode, FI_APPEND_WRITE);
set_inode_flag(inode, FI_DATA_EXIST);
folio_clear_f2fs_inline(ifolio);
f2fs_folio_put(ifolio, true);
f2fs_cache_clear_inline(ientry);
f2fs_put_cache(ientry, true);
return 0;
}
int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio)
int f2fs_recover_inline_data(struct inode *inode, struct f2fs_cached_block *entry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode *ri = NULL;
@ -308,39 +308,41 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio)
* x o -> remove data blocks, and then recover inline_data
* x x -> recover data blocks
*/
if (IS_INODE(nfolio))
ri = F2FS_INODE(nfolio);
if (IS_INODE(F2FS_I_SB(inode), entry))
ri = &CACHED_NODE(entry)->i;
if (f2fs_has_inline_data(inode) &&
ri && (ri->i_inline & F2FS_INLINE_DATA)) {
struct folio *ifolio;
struct f2fs_cached_block *ientry;
process_inline:
ifolio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_cache_wait_writeback(ientry);
src_addr = inline_data_addr(inode, nfolio);
dst_addr = inline_data_addr(inode, ifolio);
src_addr = inline_data_addr(inode, entry);
dst_addr = inline_data_addr(inode, ientry);
memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode));
set_inode_flag(inode, FI_INLINE_DATA);
set_inode_flag(inode, FI_DATA_EXIST);
folio_mark_dirty(ifolio);
f2fs_folio_put(ifolio, true);
f2fs_mark_cache_dirty(ientry);
f2fs_put_cache(ientry, true);
return 1;
}
if (f2fs_has_inline_data(inode)) {
struct folio *ifolio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
f2fs_truncate_inline_inode(inode, ifolio, 0);
struct f2fs_cached_block *ientry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
f2fs_truncate_inline_inode(inode, ientry, 0);
stat_dec_inline_inode(inode);
clear_inode_flag(inode, FI_INLINE_DATA);
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
} else if (ri && (ri->i_inline & F2FS_INLINE_DATA)) {
int ret;
@ -355,50 +357,50 @@ int f2fs_recover_inline_data(struct inode *inode, struct folio *nfolio)
struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir,
const struct f2fs_filename *fname,
struct folio **res_folio,
void **dentry_block,
bool use_hash)
{
struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb);
struct f2fs_dir_entry *de;
struct f2fs_dentry_ptr d;
struct folio *ifolio;
struct f2fs_cached_block *ientry;
void *inline_dentry;
ifolio = f2fs_get_inode_folio(sbi, dir->i_ino);
if (IS_ERR(ifolio)) {
*res_folio = ifolio;
ientry = f2fs_get_inode_cache(sbi, dir->i_ino);
if (IS_ERR(ientry)) {
*dentry_block = ientry;
return NULL;
}
inline_dentry = inline_data_addr(dir, ifolio);
inline_dentry = inline_data_addr(dir, ientry);
make_dentry_ptr_inline(dir, &d, inline_dentry);
de = f2fs_find_target_dentry(&d, fname, NULL, use_hash);
folio_unlock(ifolio);
f2fs_unlock_cache(ientry);
if (IS_ERR(de)) {
*res_folio = ERR_CAST(de);
*dentry_block = ERR_CAST(de);
de = NULL;
}
if (de)
*res_folio = ifolio;
*dentry_block = f2fs_cache_make_dentry_block(ientry);
else
f2fs_folio_put(ifolio, false);
f2fs_put_cache(ientry, false);
return de;
}
int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent,
struct folio *ifolio)
struct f2fs_cached_block *ientry)
{
struct f2fs_dentry_ptr d;
void *inline_dentry;
inline_dentry = inline_data_addr(inode, ifolio);
inline_dentry = inline_data_addr(inode, ientry);
make_dentry_ptr_inline(inode, &d, inline_dentry);
f2fs_do_make_empty_dir(inode, parent, &d);
folio_mark_dirty(ifolio);
f2fs_mark_cache_dirty(ientry);
/* update i_size to MAX_INLINE_DATA */
if (i_size_read(inode) < MAX_INLINE_DATA(inode))
@ -410,22 +412,23 @@ int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent,
* NOTE: ipage is grabbed by caller, but if any error occurs, we should
* release ipage in this function.
*/
static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio,
void *inline_dentry)
static int f2fs_move_inline_dirents(struct inode *dir,
struct f2fs_cached_block *ientry,
void *inline_dentry)
{
struct folio *folio;
struct dnode_of_data dn;
struct f2fs_dentry_block *dentry_blk;
void *dentry_blk;
struct f2fs_dentry_ptr src, dst;
int err;
folio = f2fs_grab_cache_folio(dir->i_mapping, 0, true);
if (IS_ERR(folio)) {
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
return PTR_ERR(folio);
}
set_new_dnode(&dn, dir, ifolio, NULL, 0);
set_new_dnode(&dn, dir, ientry, NULL, 0);
err = f2fs_reserve_block(&dn, 0);
if (err)
goto out;
@ -441,7 +444,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio,
goto out;
}
f2fs_folio_wait_writeback(folio, DATA, true, true);
f2fs_folio_wait_writeback(folio, true, true);
dentry_blk = folio_address(folio);
@ -449,7 +452,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio,
* Start by zeroing the full block, to ensure that all unused space is
* zeroed and no uninitialized memory is leaked to disk.
*/
memset(dentry_blk, 0, F2FS_BLKSIZE);
memset(dentry_blk, 0, F2FS_BLKSIZE(F2FS_I_SB(dir)));
make_dentry_ptr_inline(dir, &src, inline_dentry);
make_dentry_ptr_block(dir, &dst, dentry_blk);
@ -464,7 +467,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio,
folio_mark_dirty(folio);
/* clear inline dir and flag after data writeback */
f2fs_truncate_inline_inode(dir, ifolio, 0);
f2fs_truncate_inline_inode(dir, ientry, 0);
stat_dec_inline_dir(dir);
clear_inode_flag(dir, FI_INLINE_DENTRY);
@ -478,8 +481,8 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct folio *ifolio,
F2FS_I(dir)->i_inline_xattr_size = 0;
f2fs_i_depth_write(dir, 1);
if (i_size_read(dir) < PAGE_SIZE)
f2fs_i_size_write(dir, PAGE_SIZE);
if (i_size_read(dir) < F2FS_BLKSIZE(F2FS_I_SB(dir)))
f2fs_i_size_write(dir, F2FS_BLKSIZE(F2FS_I_SB(dir)));
out:
f2fs_folio_put(folio, true);
return err;
@ -544,8 +547,8 @@ static int f2fs_add_inline_entries(struct inode *dir, void *inline_dentry)
return err;
}
static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio,
void *inline_dentry)
static int f2fs_move_rehashed_dirents(struct inode *dir,
struct f2fs_cached_block *ientry, void *inline_dentry)
{
void *backup_dentry;
int err;
@ -553,20 +556,20 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio,
backup_dentry = f2fs_kmalloc(F2FS_I_SB(dir),
MAX_INLINE_DATA(dir), GFP_F2FS_ZERO);
if (!backup_dentry) {
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
return -ENOMEM;
}
memcpy(backup_dentry, inline_dentry, MAX_INLINE_DATA(dir));
f2fs_truncate_inline_inode(dir, ifolio, 0);
f2fs_truncate_inline_inode(dir, ientry, 0);
folio_unlock(ifolio);
f2fs_unlock_cache(ientry);
err = f2fs_add_inline_entries(dir, backup_dentry);
if (err)
goto recover;
folio_lock(ifolio);
f2fs_lock_cache(ientry);
stat_dec_inline_dir(dir);
clear_inode_flag(dir, FI_INLINE_DENTRY);
@ -582,31 +585,31 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct folio *ifolio,
kfree(backup_dentry);
return 0;
recover:
folio_lock(ifolio);
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_lock_cache(ientry);
f2fs_cache_wait_writeback(ientry);
memcpy(inline_dentry, backup_dentry, MAX_INLINE_DATA(dir));
f2fs_i_depth_write(dir, 0);
f2fs_i_size_write(dir, MAX_INLINE_DATA(dir));
folio_mark_dirty(ifolio);
f2fs_folio_put(ifolio, true);
f2fs_mark_cache_dirty(ientry);
f2fs_put_cache(ientry, true);
kfree(backup_dentry);
return err;
}
static int do_convert_inline_dir(struct inode *dir, struct folio *ifolio,
void *inline_dentry)
static int do_convert_inline_dir(struct inode *dir,
struct f2fs_cached_block *ientry, void *inline_dentry)
{
if (!F2FS_I(dir)->i_dir_level)
return f2fs_move_inline_dirents(dir, ifolio, inline_dentry);
return f2fs_move_inline_dirents(dir, ientry, inline_dentry);
else
return f2fs_move_rehashed_dirents(dir, ifolio, inline_dentry);
return f2fs_move_rehashed_dirents(dir, ientry, inline_dentry);
}
int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dir);
struct folio *ifolio;
struct f2fs_cached_block *ientry;
struct f2fs_filename fname;
struct f2fs_lock_context lc;
void *inline_dentry = NULL;
@ -621,22 +624,22 @@ int f2fs_try_convert_inline_dir(struct inode *dir, struct dentry *dentry)
if (err)
goto out;
ifolio = f2fs_get_inode_folio(sbi, dir->i_ino);
if (IS_ERR(ifolio)) {
err = PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(sbi, dir->i_ino);
if (IS_ERR(ientry)) {
err = PTR_ERR(ientry);
goto out_fname;
}
if (f2fs_has_enough_room(dir, ifolio, &fname)) {
f2fs_folio_put(ifolio, true);
if (f2fs_has_enough_room(dir, ientry, &fname)) {
f2fs_put_cache(ientry, true);
goto out_fname;
}
inline_dentry = inline_data_addr(dir, ifolio);
inline_dentry = inline_data_addr(dir, ientry);
err = do_convert_inline_dir(dir, ifolio, inline_dentry);
err = do_convert_inline_dir(dir, ientry, inline_dentry);
if (!err)
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
out_fname:
f2fs_free_filename(&fname);
out:
@ -648,24 +651,24 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname,
struct inode *inode, nid_t ino, umode_t mode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dir);
struct folio *ifolio;
struct f2fs_cached_block *ientry;
unsigned int bit_pos;
void *inline_dentry = NULL;
struct f2fs_dentry_ptr d;
int slots = GET_DENTRY_SLOTS(fname->disk_name.len);
struct folio *folio = NULL;
struct f2fs_cached_block *nentry = NULL;
int err = 0;
ifolio = f2fs_get_inode_folio(sbi, dir->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(sbi, dir->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
inline_dentry = inline_data_addr(dir, ifolio);
inline_dentry = inline_data_addr(dir, ientry);
make_dentry_ptr_inline(dir, &d, inline_dentry);
bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max);
if (bit_pos >= d.max) {
err = do_convert_inline_dir(dir, ifolio, inline_dentry);
err = do_convert_inline_dir(dir, ientry, inline_dentry);
if (err)
return err;
err = -EAGAIN;
@ -675,19 +678,19 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname,
if (inode) {
f2fs_down_write_nested(&F2FS_I(inode)->i_sem,
SINGLE_DEPTH_NESTING);
folio = f2fs_init_inode_metadata(inode, dir, fname, ifolio);
if (IS_ERR(folio)) {
err = PTR_ERR(folio);
nentry = f2fs_init_inode_metadata(inode, dir, fname, ientry);
if (IS_ERR(nentry)) {
err = PTR_ERR(nentry);
goto fail;
}
}
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_cache_wait_writeback(ientry);
f2fs_update_dentry(ino, mode, &d, &fname->disk_name, fname->hash,
bit_pos);
folio_mark_dirty(ifolio);
f2fs_mark_cache_dirty(ientry);
/* we don't need to mark_inode_dirty now */
if (inode) {
@ -695,9 +698,9 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname,
/* synchronize inode page's data from inode cache */
if (is_inode_flag_set(inode, FI_NEW_INODE))
f2fs_update_inode(inode, folio);
f2fs_update_inode(inode, nentry);
f2fs_folio_put(folio, true);
f2fs_put_cache(nentry, true);
}
f2fs_update_parent_metadata(dir, inode, 0);
@ -705,12 +708,12 @@ int f2fs_add_inline_entry(struct inode *dir, const struct f2fs_filename *fname,
if (inode)
f2fs_up_write(&F2FS_I(inode)->i_sem);
out:
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
return err;
}
void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry,
struct folio *folio, struct inode *dir, struct inode *inode)
struct f2fs_cached_block *ientry, struct inode *dir, struct inode *inode)
{
struct f2fs_dentry_ptr d;
void *inline_dentry;
@ -718,18 +721,18 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry,
unsigned int bit_pos;
int i;
folio_lock(folio);
f2fs_folio_wait_writeback(folio, NODE, true, true);
f2fs_lock_cache(ientry);
f2fs_cache_wait_writeback(ientry);
inline_dentry = inline_data_addr(dir, folio);
inline_dentry = inline_data_addr(dir, ientry);
make_dentry_ptr_inline(dir, &d, inline_dentry);
bit_pos = dentry - d.dentry;
for (i = 0; i < slots; i++)
__clear_bit_le(bit_pos + i, d.bitmap);
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
f2fs_mark_cache_dirty(ientry);
f2fs_put_cache(ientry, true);
inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
f2fs_mark_inode_dirty_sync(dir, true);
@ -741,21 +744,21 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry,
bool f2fs_empty_inline_dir(struct inode *dir)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dir);
struct folio *ifolio;
struct f2fs_cached_block *ientry;
unsigned int bit_pos = 2;
void *inline_dentry;
struct f2fs_dentry_ptr d;
ifolio = f2fs_get_inode_folio(sbi, dir->i_ino);
if (IS_ERR(ifolio))
ientry = f2fs_get_inode_cache(sbi, dir->i_ino);
if (IS_ERR(ientry))
return false;
inline_dentry = inline_data_addr(dir, ifolio);
inline_dentry = inline_data_addr(dir, ientry);
make_dentry_ptr_inline(dir, &d, inline_dentry);
bit_pos = find_next_bit_le(d.bitmap, d.max, bit_pos);
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
if (bit_pos < d.max)
return false;
@ -767,7 +770,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx,
struct fscrypt_str *fstr)
{
struct inode *inode = file_inode(file);
struct folio *ifolio = NULL;
struct f2fs_cached_block *ientry = NULL;
struct f2fs_dentry_ptr d;
void *inline_dentry = NULL;
int err;
@ -777,17 +780,17 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx,
if (ctx->pos == d.max)
return 0;
ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
/*
* f2fs_readdir was protected by inode.i_rwsem, it is safe to access
* ipage without page's lock held.
*/
folio_unlock(ifolio);
f2fs_unlock_cache(ientry);
inline_dentry = inline_data_addr(inode, ifolio);
inline_dentry = inline_data_addr(inode, ientry);
make_dentry_ptr_inline(inode, &d, inline_dentry);
@ -795,7 +798,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx,
if (!err)
ctx->pos = d.max;
f2fs_folio_put(ifolio, false);
f2fs_put_cache(ientry, false);
return err < 0 ? err : 0;
}
@ -806,12 +809,12 @@ int f2fs_inline_data_fiemap(struct inode *inode,
__u32 flags = FIEMAP_EXTENT_DATA_INLINE | FIEMAP_EXTENT_NOT_ALIGNED |
FIEMAP_EXTENT_LAST;
struct node_info ni;
struct folio *ifolio;
struct f2fs_cached_block *ientry;
int err = 0;
ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
if ((S_ISREG(inode->i_mode) || S_ISLNK(inode->i_mode)) &&
!f2fs_has_inline_data(inode)) {
@ -825,13 +828,13 @@ int f2fs_inline_data_fiemap(struct inode *inode,
}
if (fieinfo->fi_flags & FIEMAP_FLAG_SYNC) {
err = f2fs_write_single_node_folio(ifolio, true, false, FS_NODE_IO);
err = f2fs_write_node_cache(ientry, true, false, FS_NODE_IO);
if (err)
return err;
ifolio = f2fs_get_inode_folio(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ifolio))
return PTR_ERR(ifolio);
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
ientry = f2fs_get_inode_cache(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ientry))
return PTR_ERR(ientry);
f2fs_cache_wait_writeback(ientry);
}
ilen = min_t(size_t, MAX_INLINE_DATA(inode), i_size_read(inode));
if (start >= ilen)
@ -846,8 +849,8 @@ int f2fs_inline_data_fiemap(struct inode *inode,
if (__is_valid_data_blkaddr(ni.blk_addr)) {
byteaddr = (__u64)ni.blk_addr << inode->i_sb->s_blocksize_bits;
byteaddr += (char *)inline_data_addr(inode, ifolio) -
(char *)F2FS_INODE(ifolio);
byteaddr += (char *)inline_data_addr(inode, ientry) -
(char *)F2FS_INODE(ientry);
} else {
f2fs_bug_on(F2FS_I_SB(inode), ni.blk_addr != NEW_ADDR);
flags |= FIEMAP_EXTENT_DELALLOC | FIEMAP_EXTENT_UNKNOWN;
@ -855,6 +858,6 @@ int f2fs_inline_data_fiemap(struct inode *inode,
err = fiemap_fill_next_extent(fieinfo, start, byteaddr, ilen, flags);
trace_f2fs_fiemap(inode, start, byteaddr, ilen, flags, err);
out:
f2fs_folio_put(ifolio, true);
f2fs_put_cache(ientry, true);
return err;
}

242
fs/f2fs/inode.c
View File

@ -82,9 +82,9 @@ void f2fs_set_inode_flags(struct inode *inode)
S_ENCRYPTED|S_VERITY|S_CASEFOLD);
}
static void __get_inode_rdev(struct inode *inode, struct folio *node_folio)
static void __get_inode_rdev(struct inode *inode, struct f2fs_cached_block *ientry)
{
__le32 *addr = get_dnode_addr(inode, node_folio);
__le32 *addr = get_dnode_addr(inode, ientry);
if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode) ||
S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode)) {
@ -95,9 +95,9 @@ static void __get_inode_rdev(struct inode *inode, struct folio *node_folio)
}
}
static void __set_inode_rdev(struct inode *inode, struct folio *node_folio)
static void __set_inode_rdev(struct inode *inode, struct f2fs_cached_block *ientry)
{
__le32 *addr = get_dnode_addr(inode, node_folio);
__le32 *addr = get_dnode_addr(inode, ientry);
if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) {
if (old_valid_dev(inode->i_rdev)) {
@ -111,34 +111,33 @@ static void __set_inode_rdev(struct inode *inode, struct folio *node_folio)
}
}
static void __recover_inline_status(struct inode *inode, struct folio *ifolio)
static void __recover_inline_status(struct inode *inode, struct f2fs_cached_block *ientry)
{
void *inline_data = inline_data_addr(inode, ifolio);
void *inline_data = inline_data_addr(inode, ientry);
__le32 *start = inline_data;
__le32 *end = start + MAX_INLINE_DATA(inode) / sizeof(__le32);
while (start < end) {
if (*start++) {
f2fs_folio_wait_writeback(ifolio, NODE, true, true);
f2fs_cache_wait_writeback(ientry);
set_inode_flag(inode, FI_DATA_EXIST);
set_raw_inline(inode, F2FS_INODE(ifolio));
folio_mark_dirty(ifolio);
set_raw_inline(inode, F2FS_INODE(ientry));
f2fs_mark_cache_dirty(ientry);
return;
}
}
return;
}
static
bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio)
static bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry)
{
struct f2fs_inode *ri = &F2FS_NODE(folio)->i;
struct f2fs_inode *ri = F2FS_INODE(entry);
if (!f2fs_sb_has_inode_chksum(sbi))
return false;
if (!IS_INODE(folio) || !(ri->i_inline & F2FS_EXTRA_ATTR))
if (!IS_INODE(sbi, entry) || !(ri->i_inline & F2FS_EXTRA_ATTR))
return false;
if (!F2FS_FITS_IN_INODE(ri, le16_to_cpu(ri->i_extra_isize),
@ -148,11 +147,10 @@ bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio)
return true;
}
static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio)
static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry)
{
struct f2fs_node *node = F2FS_NODE(folio);
struct f2fs_inode *ri = &node->i;
__le32 ino = node->footer.ino;
struct f2fs_inode *ri = F2FS_INODE(entry);
__le32 ino = F2FS_NODE_FOOTER(sbi, entry)->ino;
__le32 gen = ri->i_generation;
__u32 chksum, chksum_seed;
__u32 dummy_cs = 0;
@ -166,11 +164,11 @@ static __u32 f2fs_inode_chksum(struct f2fs_sb_info *sbi, struct folio *folio)
chksum = f2fs_chksum(chksum, (__u8 *)&dummy_cs, cs_size);
offset += cs_size;
chksum = f2fs_chksum(chksum, (__u8 *)ri + offset,
F2FS_BLKSIZE - offset);
F2FS_BLKSIZE(sbi) - offset);
return chksum;
}
bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio)
bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry)
{
struct f2fs_inode *ri;
__u32 provided, calculated;
@ -178,35 +176,33 @@ bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct folio *folio)
if (unlikely(is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN)))
return true;
#ifdef CONFIG_F2FS_CHECK_FS
if (!f2fs_enable_inode_chksum(sbi, folio))
#else
if (!f2fs_enable_inode_chksum(sbi, folio) ||
folio_test_dirty(folio) ||
folio_test_writeback(folio))
#endif
if (!f2fs_enable_inode_chksum(sbi, entry))
return true;
#ifndef CONFIG_F2FS_CHECK_FS
if (f2fs_cache_test_dirty(entry) || f2fs_cache_test_writeback(entry))
return true;
#endif
ri = &F2FS_NODE(folio)->i;
ri = F2FS_INODE(entry);
provided = le32_to_cpu(ri->i_inode_checksum);
calculated = f2fs_inode_chksum(sbi, folio);
calculated = f2fs_inode_chksum(sbi, entry);
if (provided != calculated)
f2fs_warn(sbi, "checksum invalid, nid = %lu, ino_of_node = %x, %x vs. %x",
folio->index, ino_of_node(folio),
f2fs_warn(sbi, "checksum invalid, nid = %lu, ino_of_node = %u, %x vs. %x",
entry->index, ino_of_node(sbi, entry),
provided, calculated);
return provided == calculated;
}
void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct folio *folio)
void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct f2fs_cached_block *entry)
{
struct f2fs_inode *ri = &F2FS_NODE(folio)->i;
struct f2fs_inode *ri = F2FS_INODE(entry);
if (!f2fs_enable_inode_chksum(sbi, folio))
if (!f2fs_enable_inode_chksum(sbi, entry))
return;
ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, folio));
ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, entry));
}
static bool sanity_check_compress_inode(struct inode *inode,
@ -222,11 +218,11 @@ static bool sanity_check_compress_inode(struct inode *inode,
return false;
}
if (le64_to_cpu(ri->i_compr_blocks) >
SECTOR_TO_BLOCK(inode->i_blocks)) {
SECTOR_TO_BLOCK(sbi, inode->i_blocks)) {
f2fs_warn(sbi,
"%s: inode (ino=%llx) has inconsistent i_compr_blocks:%llu, i_blocks:%llu, run fsck to fix",
__func__, inode->i_ino, le64_to_cpu(ri->i_compr_blocks),
SECTOR_TO_BLOCK(inode->i_blocks));
SECTOR_TO_BLOCK(sbi, inode->i_blocks));
return false;
}
if (ri->i_log_cluster_size < MIN_COMPRESS_LOG_SIZE ||
@ -281,28 +277,29 @@ static bool sanity_check_compress_inode(struct inode *inode,
return false;
}
static bool sanity_check_inode(struct inode *inode, struct folio *node_folio)
static bool sanity_check_inode(struct inode *inode,
struct f2fs_cached_block *node_entry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
struct f2fs_inode *ri = F2FS_INODE(node_folio);
struct f2fs_inode *ri = F2FS_INODE(node_entry);
unsigned long long iblocks;
iblocks = le64_to_cpu(F2FS_INODE(node_folio)->i_blocks);
iblocks = le64_to_cpu(ri->i_blocks);
if (!iblocks) {
f2fs_warn(sbi, "%s: corrupted inode i_blocks i_ino=%llx iblocks=%llu, run fsck to fix.",
__func__, inode->i_ino, iblocks);
return false;
}
if (ino_of_node(node_folio) != nid_of_node(node_folio)) {
if (ino_of_node(sbi, node_entry) != nid_of_node(sbi, node_entry)) {
f2fs_warn(sbi, "%s: corrupted inode footer i_ino=%llx, ino,nid: [%u, %u] run fsck to fix.",
__func__, inode->i_ino,
ino_of_node(node_folio), nid_of_node(node_folio));
ino_of_node(sbi, node_entry), nid_of_node(sbi, node_entry));
return false;
}
if (ino_of_node(node_folio) == fi->i_xattr_nid) {
if (ino_of_node(sbi, node_entry) == fi->i_xattr_nid) {
f2fs_warn(sbi, "%s: corrupted inode i_ino=%llx, xnid=%x, run fsck to fix.",
__func__, inode->i_ino, fi->i_xattr_nid);
return false;
@ -338,12 +335,13 @@ static bool sanity_check_inode(struct inode *inode, struct folio *node_folio)
}
if (f2fs_sb_has_flexible_inline_xattr(sbi) &&
(fi->i_inline_xattr_size > MAX_INLINE_XATTR_SIZE ||
(fi->i_inline_xattr_size > MAX_INLINE_XATTR_SIZE(i_blocksize(inode)) ||
(f2fs_has_inline_xattr(inode) &&
fi->i_inline_xattr_size < MIN_INLINE_XATTR_SIZE))) {
f2fs_warn(sbi, "%s: inode (ino=%llx) has corrupted i_inline_xattr_size: %d, min: %zu, max: %lu",
f2fs_warn(sbi, "%s: inode (ino=%llx) has corrupted i_inline_xattr_size: %d, min: %zu, max: %zu",
__func__, inode->i_ino, fi->i_inline_xattr_size,
MIN_INLINE_XATTR_SIZE, MAX_INLINE_XATTR_SIZE);
MIN_INLINE_XATTR_SIZE,
(size_t)MAX_INLINE_XATTR_SIZE(i_blocksize(inode)));
return false;
}
@ -375,7 +373,7 @@ static bool sanity_check_inode(struct inode *inode, struct folio *node_folio)
}
}
if (f2fs_sanity_check_inline_data(inode, node_folio)) {
if (f2fs_sanity_check_inline_data(inode, node_entry)) {
f2fs_warn(sbi, "%s: inode (ino=%llx, mode=%u) should not have inline_data, run fsck to fix",
__func__, inode->i_ino, inode->i_mode);
return false;
@ -428,7 +426,7 @@ static int do_read_inode(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
struct folio *node_folio;
struct f2fs_cached_block *node_entry;
struct f2fs_inode *ri;
projid_t i_projid;
@ -436,18 +434,19 @@ static int do_read_inode(struct inode *inode)
if (f2fs_check_nid_range(sbi, inode->i_ino))
return -EINVAL;
node_folio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(node_folio))
return PTR_ERR(node_folio);
node_entry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(node_entry))
return PTR_ERR(node_entry);
ri = F2FS_INODE(node_folio);
ri = F2FS_INODE(node_entry);
inode->i_mode = le16_to_cpu(ri->i_mode);
i_uid_write(inode, le32_to_cpu(ri->i_uid));
i_gid_write(inode, le32_to_cpu(ri->i_gid));
set_nlink(inode, le32_to_cpu(ri->i_links));
inode->i_size = le64_to_cpu(ri->i_size);
inode->i_blocks = SECTOR_FROM_BLOCK(le64_to_cpu(ri->i_blocks) - 1);
inode->i_blocks = SECTOR_FROM_BLOCK(sbi,
le64_to_cpu(ri->i_blocks) - 1);
inode_set_atime(inode, le64_to_cpu(ri->i_atime),
le32_to_cpu(ri->i_atime_nsec));
@ -490,8 +489,8 @@ static int do_read_inode(struct inode *inode)
fi->i_inline_xattr_size = 0;
}
if (!sanity_check_inode(inode, node_folio)) {
f2fs_folio_put(node_folio, true);
if (!sanity_check_inode(inode, node_entry)) {
f2fs_put_cache(node_entry, true);
set_sbi_flag(sbi, SBI_NEED_FSCK);
f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE);
fserror_report_file_metadata(inode, -EFSCORRUPTED, GFP_NOFS);
@ -500,17 +499,17 @@ static int do_read_inode(struct inode *inode)
/* check data exist */
if (f2fs_has_inline_data(inode) && !f2fs_exist_data(inode))
__recover_inline_status(inode, node_folio);
__recover_inline_status(inode, node_entry);
/* try to recover cold bit for non-dir inode */
if (!S_ISDIR(inode->i_mode) && !is_cold_node(node_folio)) {
f2fs_folio_wait_writeback(node_folio, NODE, true, true);
set_cold_node(node_folio, false);
folio_mark_dirty(node_folio);
if (!S_ISDIR(inode->i_mode) && !is_cold_node(sbi, node_entry)) {
f2fs_cache_wait_writeback(node_entry);
set_cold_node(sbi, node_entry, false);
f2fs_mark_cache_dirty(node_entry);
}
/* get rdev by using inline_info */
__get_inode_rdev(inode, node_folio);
__get_inode_rdev(inode, node_entry);
if (!f2fs_need_inode_block_update(sbi, inode->i_ino))
fi->last_disk_size = inode->i_size;
@ -553,18 +552,18 @@ static int do_read_inode(struct inode *inode)
init_idisk_time(inode);
if (!sanity_check_extent_cache(inode, node_folio)) {
f2fs_folio_put(node_folio, true);
if (!sanity_check_extent_cache(inode, node_entry)) {
f2fs_put_cache(node_entry, true);
f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE);
fserror_report_file_metadata(inode, -EFSCORRUPTED, GFP_NOFS);
return -EFSCORRUPTED;
}
/* Need all the flag bits */
f2fs_init_read_extent_tree(inode, node_folio);
f2fs_init_read_extent_tree(inode, node_entry);
f2fs_init_age_extent_tree(inode);
f2fs_folio_put(node_folio, true);
f2fs_put_cache(node_entry, true);
stat_inc_inline_xattr(inode);
stat_inc_inline_inode(inode);
@ -575,17 +574,6 @@ static int do_read_inode(struct inode *inode)
return 0;
}
static bool is_meta_ino(struct f2fs_sb_info *sbi, unsigned int ino)
{
if (ino == F2FS_NODE_INO(sbi) || ino == F2FS_META_INO(sbi))
return true;
#ifdef CONFIG_F2FS_FS_COMPRESSION
if (test_opt(sbi, COMPRESS_CACHE) && ino == F2FS_COMPRESS_INO(sbi))
return true;
#endif
return false;
}
struct inode *f2fs_iget(struct super_block *sb, unsigned long ino)
{
struct f2fs_sb_info *sbi = F2FS_SB(sb);
@ -597,48 +585,17 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino)
return ERR_PTR(-ENOMEM);
if (!(inode_state_read_once(inode) & I_NEW)) {
if (is_meta_ino(sbi, ino)) {
f2fs_err(sbi, "inaccessible inode: %lu, run fsck to repair", ino);
set_sbi_flag(sbi, SBI_NEED_FSCK);
ret = -EFSCORRUPTED;
trace_f2fs_iget_exit(inode, ret);
iput(inode);
f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE);
fserror_report_file_metadata(inode, ret, GFP_NOFS);
return ERR_PTR(ret);
}
trace_f2fs_iget(inode);
return inode;
}
if (is_meta_ino(sbi, ino))
goto make_now;
ret = do_read_inode(inode);
if (ret)
goto bad_inode;
make_now:
f2fs_set_inode_flags(inode);
if (ino == F2FS_NODE_INO(sbi)) {
inode->i_mapping->a_ops = &f2fs_node_aops;
mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS);
} else if (ino == F2FS_META_INO(sbi)) {
inode->i_mapping->a_ops = &f2fs_meta_aops;
mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS);
} else if (ino == F2FS_COMPRESS_INO(sbi)) {
#ifdef CONFIG_F2FS_FS_COMPRESSION
inode->i_mapping->a_ops = &f2fs_compress_aops;
/*
* generic_error_remove_folio only truncates pages of regular
* inode
*/
inode->i_mode |= S_IFREG;
#endif
mapping_set_gfp_mask(inode->i_mapping,
GFP_NOFS | __GFP_HIGHMEM | __GFP_MOVABLE);
} else if (S_ISREG(inode->i_mode)) {
if (S_ISREG(inode->i_mode)) {
inode->i_op = &f2fs_file_inode_operations;
inode->i_fop = &f2fs_file_operations;
inode->i_mapping->a_ops = &f2fs_dblock_aops;
@ -694,25 +651,28 @@ struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino)
return inode;
}
void f2fs_update_inode(struct inode *inode, struct folio *node_folio)
void f2fs_update_inode(struct inode *inode,
struct f2fs_cached_block *node_entry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
struct f2fs_inode *ri;
struct extent_tree *et = fi->extent_tree[EX_READ];
f2fs_folio_wait_writeback(node_folio, NODE, true, true);
folio_mark_dirty(node_folio);
f2fs_cache_wait_writeback(node_entry);
f2fs_mark_cache_dirty(node_entry);
f2fs_inode_synced(inode);
ri = F2FS_INODE(node_folio);
ri = F2FS_INODE(node_entry);
ri->i_mode = cpu_to_le16(inode->i_mode);
ri->i_advise = fi->i_advise;
ri->i_uid = cpu_to_le32(i_uid_read(inode));
ri->i_gid = cpu_to_le32(i_gid_read(inode));
ri->i_links = cpu_to_le32(inode->i_nlink);
ri->i_blocks = cpu_to_le64(SECTOR_TO_BLOCK(READ_ONCE(inode->i_blocks)) + 1);
ri->i_blocks = cpu_to_le64(SECTOR_TO_BLOCK(sbi,
READ_ONCE(inode->i_blocks)) + 1);
if (!f2fs_is_atomic_file(inode) ||
is_inode_flag_set(inode, FI_ATOMIC_COMMITTED))
@ -780,27 +740,27 @@ void f2fs_update_inode(struct inode *inode, struct folio *node_folio)
}
}
__set_inode_rdev(inode, node_folio);
__set_inode_rdev(inode, node_entry);
/* deleted inode */
if (inode->i_nlink == 0)
folio_clear_f2fs_inline(node_folio);
f2fs_cache_clear_inline(node_entry);
init_idisk_time(inode);
#ifdef CONFIG_F2FS_CHECK_FS
f2fs_inode_chksum_set(F2FS_I_SB(inode), node_folio);
f2fs_inode_chksum_set(F2FS_I_SB(inode), node_entry);
#endif
}
void f2fs_update_inode_page(struct inode *inode)
void f2fs_update_inode_cache(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct folio *node_folio;
struct f2fs_cached_block *node_entry;
int count = 0;
retry:
node_folio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(node_folio)) {
int err = PTR_ERR(node_folio);
node_entry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(node_entry)) {
int err = PTR_ERR(node_entry);
/* The node block was truncated. */
if (err == -ENOENT)
@ -816,18 +776,14 @@ void f2fs_update_inode_page(struct inode *inode)
f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_UPDATE_INODE);
return;
}
f2fs_update_inode(inode, node_folio);
f2fs_folio_put(node_folio, true);
f2fs_update_inode(inode, node_entry);
f2fs_put_cache(node_entry, true);
}
int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
if (inode->i_ino == F2FS_NODE_INO(sbi) ||
inode->i_ino == F2FS_META_INO(sbi))
return 0;
/*
* atime could be updated without dirtying f2fs inode in lazytime mode
*/
@ -851,7 +807,7 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc)
* We need to balance fs here to prevent from producing dirty node pages
* during the urgent cleaning time when running out of free sections.
*/
f2fs_update_inode_page(inode);
f2fs_update_inode_cache(inode);
if (wbc && wbc->nr_to_write)
f2fs_balance_fs(sbi, true);
return 0;
@ -892,7 +848,7 @@ static void f2fs_evict_inode_work(struct work_struct *work)
/*
* Return true, if we shouldn't go through post_evict_inode.
*/
static bool f2fs_pre_evict_inode(struct inode *inode)
static void f2fs_pre_evict_inode(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
@ -918,19 +874,12 @@ static bool f2fs_pre_evict_inode(struct inode *inode)
test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode))
f2fs_invalidate_compress_pages(sbi, inode->i_ino);
if (inode->i_ino == F2FS_NODE_INO(sbi) ||
inode->i_ino == F2FS_META_INO(sbi) ||
inode->i_ino == F2FS_COMPRESS_INO(sbi))
return true;
f2fs_bug_on(sbi, get_dirty_pages(inode));
f2fs_remove_dirty_inode(inode);
f2fs_remove_donate_inode(inode);
if (!IS_DEVICE_ALIASING(inode))
f2fs_destroy_extent_tree(inode);
return false;
}
static void f2fs_delete_inode(struct inode *inode)
@ -964,7 +913,7 @@ static void f2fs_delete_inode(struct inode *inode)
goto error_check;
f2fs_lock_op(sbi, &lc);
err = f2fs_remove_inode_page(inode);
err = f2fs_remove_inode_cache(inode);
f2fs_unlock_op(sbi, &lc);
if (err == -ENOENT) {
@ -996,13 +945,13 @@ static void f2fs_delete_inode(struct inode *inode)
if (!err)
goto unfreeze_out;
f2fs_update_inode_page(inode);
f2fs_update_inode_cache(inode);
if (dquot_initialize_needed(inode))
set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
/*
* If both f2fs_truncate() and f2fs_update_inode_page() failed
* If both f2fs_truncate() and f2fs_update_inode_cache() failed
* due to fuzzed corrupted inode, call f2fs_inode_synced() to
* avoid triggering later f2fs_bug_on().
*/
@ -1046,18 +995,17 @@ static void f2fs_post_evict_inode(struct inode *inode)
/* for the case f2fs_new_inode() was failed, .i_ino is zero, skip it */
if (inode->i_ino)
invalidate_mapping_pages(NODE_MAPPING(sbi), inode->i_ino,
inode->i_ino);
f2fs_invalidate_node_cache(sbi, inode->i_ino);
if (xnid)
invalidate_mapping_pages(NODE_MAPPING(sbi), xnid, xnid);
f2fs_invalidate_node_cache(sbi, xnid);
if (!inode->i_nlink)
goto skip_record;
if (is_inode_flag_set(inode, FI_APPEND_WRITE))
record_bits = BIT(APPEND_INO);
record_bits |= BIT(APPEND_INO);
if (is_inode_flag_set(inode, FI_UPDATE_WRITE))
record_bits = BIT(UPDATE_INO);
record_bits |= BIT(UPDATE_INO);
if (!record_bits)
goto skip_record;
@ -1093,15 +1041,13 @@ static void f2fs_post_evict_inode(struct inode *inode)
*/
void f2fs_evict_inode(struct inode *inode)
{
if (f2fs_pre_evict_inode(inode))
goto clear_out;
f2fs_pre_evict_inode(inode);
if (!inode->i_nlink && !is_bad_inode(inode))
f2fs_delete_inode(inode);
f2fs_post_evict_inode(inode);
clear_out:
fscrypt_put_encryption_info(inode);
clear_inode(inode);
}
@ -1124,7 +1070,7 @@ void f2fs_handle_failed_inode(struct inode *inode,
* we must call this to avoid inode being remained as dirty, resulting
* in a panic when flushing dirty inodes in gdirty_list.
*/
f2fs_update_inode_page(inode);
f2fs_update_inode_cache(inode);
f2fs_inode_synced(inode);
/* don't make bad inode, since it becomes a regular file. */

11
fs/f2fs/iostat.h
View File

@ -60,6 +60,13 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio)
return iostat_ctx->post_read_ctx;
}
static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx)
{
struct bio_iostat_ctx *iostat_ctx = bio->bi_private;
iostat_ctx->post_read_ctx = ctx;
}
extern void iostat_update_and_unbind_ctx(struct bio *bio);
extern void iostat_alloc_and_bind_ctx(struct f2fs_sb_info *sbi,
struct bio *bio, struct bio_post_read_ctx *ctx);
@ -81,6 +88,10 @@ static inline struct bio_post_read_ctx *get_post_read_ctx(struct bio *bio)
{
return bio->bi_private;
}
static inline void iostat_set_post_read_ctx(struct bio *bio, void *ctx)
{
bio->bi_private = ctx;
}
static inline int f2fs_init_iostat_processing(void) { return 0; }
static inline void f2fs_destroy_iostat_processing(void) {}
static inline int f2fs_init_iostat(struct f2fs_sb_info *sbi) { return 0; }

114
fs/f2fs/namei.c
View File

@ -473,12 +473,13 @@ static int f2fs_link(struct dentry *old_dentry, struct inode *dir,
struct dentry *f2fs_get_parent(struct dentry *child)
{
struct folio *folio;
unsigned long ino = f2fs_inode_by_name(d_inode(child), &dotdot_name, &folio);
void *dentry_block = NULL;
unsigned long ino = f2fs_inode_by_name(d_inode(child),
&dotdot_name, &dentry_block);
if (!ino) {
if (IS_ERR(folio))
return ERR_CAST(folio);
if (IS_ERR(dentry_block))
return ERR_CAST(dentry_block);
return ERR_PTR(-ENOENT);
}
return d_obtain_alias(f2fs_iget(child->d_sb, ino));
@ -489,7 +490,7 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry,
{
struct inode *inode = NULL;
struct f2fs_dir_entry *de;
struct folio *folio;
void *dentry_block = NULL;
struct dentry *new;
nid_t ino = -1;
int err = 0;
@ -507,12 +508,12 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry,
goto out_splice;
if (err)
goto out;
de = __f2fs_find_entry(dir, &fname, &folio);
de = __f2fs_find_entry(dir, &fname, &dentry_block);
f2fs_free_filename(&fname);
if (!de) {
if (IS_ERR(folio)) {
err = PTR_ERR(folio);
if (IS_ERR(dentry_block)) {
err = PTR_ERR(dentry_block);
goto out;
}
err = -ENOENT;
@ -520,7 +521,7 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry,
}
ino = le32_to_cpu(de->ino);
f2fs_folio_put(folio, false);
f2fs_put_dentry_block(dentry_block, false);
inode = f2fs_iget(dir->i_sb, ino);
if (IS_ERR(inode)) {
@ -572,7 +573,7 @@ static int __do_unlink(struct inode *dir, struct inode *inode,
struct f2fs_sb_info *sbi = F2FS_I_SB(dir);
struct f2fs_dir_entry *de;
struct f2fs_lock_context lc;
struct folio *folio;
void *dentry_block = NULL;
int err;
if (IS_DEVICE_ALIASING(inode))
@ -588,9 +589,9 @@ static int __do_unlink(struct inode *dir, struct inode *inode,
if (err)
return err;
de = f2fs_find_entry(dir, name, &folio);
de = f2fs_find_entry(dir, name, &dentry_block);
if (!de)
return IS_ERR(folio) ? PTR_ERR(folio) : 0;
return IS_ERR(dentry_block) ? PTR_ERR(dentry_block) : 0;
if (unlikely(inode->i_nlink == 0)) {
f2fs_warn(sbi, "%s: inode (ino=%llx) has zero i_nlink",
@ -610,7 +611,7 @@ static int __do_unlink(struct inode *dir, struct inode *inode,
f2fs_unlock_op(sbi, &lc);
goto err_out;
}
f2fs_delete_entry(de, folio, dir, inode);
f2fs_delete_entry(de, dentry_block, dir, inode);
f2fs_unlock_op(sbi, &lc);
return 0;
@ -618,7 +619,7 @@ static int __do_unlink(struct inode *dir, struct inode *inode,
err = -EFSCORRUPTED;
set_sbi_flag(sbi, SBI_NEED_FSCK);
err_out:
f2fs_folio_put(folio, false);
f2fs_put_dentry_block(dentry_block, false);
return err;
}
@ -967,8 +968,9 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
struct inode *old_inode = d_inode(old_dentry);
struct inode *new_inode = d_inode(new_dentry);
struct inode *whiteout = NULL;
struct folio *old_dir_folio = NULL;
struct folio *old_folio, *new_folio = NULL;
void *old_dir_block = NULL;
void *old_block = NULL;
void *new_block = NULL;
struct f2fs_dir_entry *old_dir_entry = NULL;
struct f2fs_dir_entry *old_entry;
struct f2fs_dir_entry *new_entry;
@ -1032,18 +1034,18 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
}
err = -ENOENT;
old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_folio);
old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_block);
if (!old_entry) {
if (IS_ERR(old_folio))
err = PTR_ERR(old_folio);
if (IS_ERR(old_block))
err = PTR_ERR(old_block);
goto out;
}
if (old_is_dir && old_dir != new_dir) {
old_dir_entry = f2fs_parent_dir(old_inode, &old_dir_folio);
old_dir_entry = f2fs_parent_dir(old_inode, &old_dir_block);
if (!old_dir_entry) {
if (IS_ERR(old_dir_folio))
err = PTR_ERR(old_dir_folio);
if (IS_ERR(old_dir_block))
err = PTR_ERR(old_dir_block);
goto out_old;
}
}
@ -1060,10 +1062,10 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
err = -ENOENT;
new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name,
&new_folio);
&new_block);
if (!new_entry) {
if (IS_ERR(new_folio))
err = PTR_ERR(new_folio);
if (IS_ERR(new_block))
err = PTR_ERR(new_block);
goto out_dir;
}
@ -1075,8 +1077,8 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
if (err)
goto put_out_dir;
f2fs_set_link(new_dir, new_entry, new_folio, old_inode);
new_folio = NULL;
f2fs_set_link(new_dir, new_entry, new_block, old_inode);
new_block = NULL;
inode_set_ctime_current(new_inode);
f2fs_down_write(&F2FS_I(new_inode)->i_sem);
@ -1115,8 +1117,8 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
inode_set_ctime_current(old_inode);
f2fs_mark_inode_dirty_sync(old_inode, true);
f2fs_delete_entry(old_entry, old_folio, old_dir, NULL);
old_folio = NULL;
f2fs_delete_entry(old_entry, old_block, old_dir, NULL);
old_block = NULL;
if (whiteout) {
set_inode_flag(whiteout, FI_INC_LINK);
@ -1134,7 +1136,7 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
}
if (old_dir_entry)
f2fs_set_link(old_inode, old_dir_entry, old_dir_folio, new_dir);
f2fs_set_link(old_inode, old_dir_entry, old_dir_block, new_dir);
if (old_is_dir)
f2fs_i_links_write(old_dir, false);
@ -1158,12 +1160,12 @@ static int f2fs_rename(const struct mnt_idmap *idmap, struct inode *old_dir,
put_out_dir:
f2fs_unlock_op(sbi, &lc);
f2fs_folio_put(new_folio, false);
f2fs_put_dentry_block(new_block, false);
out_dir:
if (old_dir_entry)
f2fs_folio_put(old_dir_folio, false);
f2fs_put_dentry_block(old_dir_block, false);
out_old:
f2fs_folio_put(old_folio, false);
f2fs_put_dentry_block(old_block, false);
out:
iput(whiteout);
return err;
@ -1175,8 +1177,10 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
struct f2fs_sb_info *sbi = F2FS_I_SB(old_dir);
struct inode *old_inode = d_inode(old_dentry);
struct inode *new_inode = d_inode(new_dentry);
struct folio *old_dir_folio, *new_dir_folio;
struct folio *old_folio, *new_folio;
void *old_dir_block = NULL;
void *new_dir_block = NULL;
void *old_block = NULL;
void *new_block = NULL;
struct f2fs_dir_entry *old_dir_entry = NULL, *new_dir_entry = NULL;
struct f2fs_dir_entry *old_entry, *new_entry;
struct f2fs_lock_context lc;
@ -1208,17 +1212,17 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
goto out;
err = -ENOENT;
old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_folio);
old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_block);
if (!old_entry) {
if (IS_ERR(old_folio))
err = PTR_ERR(old_folio);
if (IS_ERR(old_block))
err = PTR_ERR(old_block);
goto out;
}
new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, &new_folio);
new_entry = f2fs_find_entry(new_dir, &new_dentry->d_name, &new_block);
if (!new_entry) {
if (IS_ERR(new_folio))
err = PTR_ERR(new_folio);
if (IS_ERR(new_block))
err = PTR_ERR(new_block);
goto out_old;
}
@ -1226,20 +1230,20 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
if (old_dir != new_dir) {
if (S_ISDIR(old_inode->i_mode)) {
old_dir_entry = f2fs_parent_dir(old_inode,
&old_dir_folio);
&old_dir_block);
if (!old_dir_entry) {
if (IS_ERR(old_dir_folio))
err = PTR_ERR(old_dir_folio);
if (IS_ERR(old_dir_block))
err = PTR_ERR(old_dir_block);
goto out_new;
}
}
if (S_ISDIR(new_inode->i_mode)) {
new_dir_entry = f2fs_parent_dir(new_inode,
&new_dir_folio);
&new_dir_block);
if (!new_dir_entry) {
if (IS_ERR(new_dir_folio))
err = PTR_ERR(new_dir_folio);
if (IS_ERR(new_dir_block))
err = PTR_ERR(new_dir_block);
goto out_old_dir;
}
}
@ -1266,14 +1270,14 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
/* update ".." directory entry info of old dentry */
if (old_dir_entry)
f2fs_set_link(old_inode, old_dir_entry, old_dir_folio, new_dir);
f2fs_set_link(old_inode, old_dir_entry, old_dir_block, new_dir);
/* update ".." directory entry info of new dentry */
if (new_dir_entry)
f2fs_set_link(new_inode, new_dir_entry, new_dir_folio, old_dir);
f2fs_set_link(new_inode, new_dir_entry, new_dir_block, old_dir);
/* update directory entry info of old dir inode */
f2fs_set_link(old_dir, old_entry, old_folio, new_inode);
f2fs_set_link(old_dir, old_entry, old_block, new_inode);
f2fs_down_write(&F2FS_I(old_inode)->i_sem);
if (!old_dir_entry)
@ -1292,7 +1296,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
f2fs_mark_inode_dirty_sync(old_dir, true);
/* update directory entry info of new dir inode */
f2fs_set_link(new_dir, new_entry, new_folio, old_inode);
f2fs_set_link(new_dir, new_entry, new_block, old_inode);
f2fs_down_write(&F2FS_I(new_inode)->i_sem);
if (!new_dir_entry)
@ -1327,16 +1331,16 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
return 0;
out_new_dir:
if (new_dir_entry) {
f2fs_folio_put(new_dir_folio, false);
f2fs_put_dentry_block(new_dir_block, false);
}
out_old_dir:
if (old_dir_entry) {
f2fs_folio_put(old_dir_folio, false);
f2fs_put_dentry_block(old_dir_block, false);
}
out_new:
f2fs_folio_put(new_folio, false);
f2fs_put_dentry_block(new_block, false);
out_old:
f2fs_folio_put(old_folio, false);
f2fs_put_dentry_block(old_block, false);
out:
return err;
}

1343
fs/f2fs/node.c
View File

File diff suppressed because it is too large Load Diff

193
fs/f2fs/node.h
View File

@ -6,19 +6,26 @@
* http://www.samsung.com/
*/
/* start node id of a node block dedicated to the given node id */
#define START_NID(nid) (((nid) / NAT_ENTRY_PER_BLOCK) * NAT_ENTRY_PER_BLOCK)
static inline nid_t f2fs_start_nid(struct f2fs_sb_info *sbi, nid_t nid)
{
unsigned int entries = NAT_ENTRY_PER_BLOCK(sbi);
return (nid / entries) * entries;
}
/* node block offset on the NAT area dedicated to the given start node id */
#define NAT_BLOCK_OFFSET(start_nid) ((start_nid) / NAT_ENTRY_PER_BLOCK)
#define NAT_BLOCK_OFFSET(sbi, start_nid) \
((start_nid) / NAT_ENTRY_PER_BLOCK(sbi))
/* # of pages to perform synchronous readahead before building free nids */
#define FREE_NID_PAGES 8
#define MAX_FREE_NIDS (NAT_ENTRY_PER_BLOCK * FREE_NID_PAGES)
/* # of blocks to perform synchronous readahead before building free nids */
#define FREE_NID_BLOCKS 8
#define MAX_FREE_NIDS(sbi) ((unsigned long)NAT_ENTRY_PER_BLOCK(sbi) * \
FREE_NID_BLOCKS)
/* size of free nid batch when shrinking */
#define SHRINK_NID_BATCH_SIZE 8
#define DEF_RA_NID_PAGES 0 /* # of nid pages to be readaheaded */
#define DEF_RA_NID_BLOCKS 0 /* # of nid blocks to be readaheaded */
/* maximum readahead size for node during getting data blocks */
#define MAX_RA_NODE 128
@ -37,8 +44,8 @@
/* vector size for gang look-up from nat cache that consists of radix tree */
#define NAT_VEC_SIZE 32
/* return value for read_node_page */
#define LOCKED_PAGE 1
/* return value for read_node_cache */
#define LOCKED_CACHE 1
/* check pinned file's alignment status of physical blocks */
#define FILE_NOT_ALIGNED 1
@ -150,7 +157,9 @@ enum mem_type {
READ_EXTENT_CACHE, /* indicates read extent cache */
AGE_EXTENT_CACHE, /* indicates age extent cache */
DISCARD_CACHE, /* indicates memory of cached discard cmds */
COMPRESS_PAGE, /* indicates memory of cached compressed pages */
COMPRESS_BLOCK, /* indicates memory of cached compressed blocks */
META_BLOCK, /* indicates memory of cached meta blocks */
NODE_BLOCK, /* indicates memory of cached node blocks */
BASE_CHECK, /* check kernel status */
};
@ -208,7 +217,7 @@ static inline pgoff_t current_nat_addr(struct f2fs_sb_info *sbi, nid_t start)
* OLD = (segment_off * 512) * 2 + off_in_segment
* NEW = 2 * (segment_off * 512 + off_in_segment) - off_in_segment
*/
block_off = NAT_BLOCK_OFFSET(start);
block_off = NAT_BLOCK_OFFSET(sbi, start);
block_addr = (pgoff_t)(nm_i->nat_blkaddr +
(block_off << 1) -
@ -230,9 +239,10 @@ static inline pgoff_t next_nat_addr(struct f2fs_sb_info *sbi,
return block_addr + nm_i->nat_blkaddr;
}
static inline void set_to_next_nat(struct f2fs_nm_info *nm_i, nid_t start_nid)
static inline void set_to_next_nat(struct f2fs_sb_info *sbi,
struct f2fs_nm_info *nm_i, nid_t start_nid)
{
unsigned int block_off = NAT_BLOCK_OFFSET(start_nid);
unsigned int block_off = NAT_BLOCK_OFFSET(sbi, start_nid);
f2fs_change_bit(block_off, nm_i->nat_bitmap);
#ifdef CONFIG_F2FS_CHECK_FS
@ -240,90 +250,95 @@ static inline void set_to_next_nat(struct f2fs_nm_info *nm_i, nid_t start_nid)
#endif
}
static inline nid_t ino_of_node(const struct folio *node_folio)
static inline nid_t ino_of_node(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
struct f2fs_node *rn = F2FS_NODE(node_folio);
return le32_to_cpu(rn->footer.ino);
return le32_to_cpu(F2FS_NODE_FOOTER(sbi, entry)->ino);
}
static inline nid_t nid_of_node(const struct folio *node_folio)
static inline nid_t nid_of_node(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
struct f2fs_node *rn = F2FS_NODE(node_folio);
return le32_to_cpu(rn->footer.nid);
return le32_to_cpu(F2FS_NODE_FOOTER(sbi, entry)->nid);
}
static inline unsigned int ofs_of_node(const struct folio *node_folio)
static inline unsigned int ofs_of_node(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
struct f2fs_node *rn = F2FS_NODE(node_folio);
unsigned flag = le32_to_cpu(rn->footer.flag);
unsigned int flag = le32_to_cpu(F2FS_NODE_FOOTER(sbi, entry)->flag);
return flag >> OFFSET_BIT_SHIFT;
}
static inline __u64 cpver_of_node(const struct folio *node_folio)
static inline __u64 cpver_of_node(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
struct f2fs_node *rn = F2FS_NODE(node_folio);
return le64_to_cpu(rn->footer.cp_ver);
return le64_to_cpu(F2FS_NODE_FOOTER(sbi, entry)->cp_ver);
}
static inline block_t next_blkaddr_of_node(const struct folio *node_folio)
static inline block_t next_blkaddr_of_node(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
struct f2fs_node *rn = F2FS_NODE(node_folio);
return le32_to_cpu(rn->footer.next_blkaddr);
return le32_to_cpu(F2FS_NODE_FOOTER(sbi, entry)->next_blkaddr);
}
static inline void fill_node_footer(const struct folio *folio, nid_t nid,
static inline void fill_node_footer(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, nid_t nid,
nid_t ino, unsigned int ofs, bool reset)
{
struct f2fs_node *rn = F2FS_NODE(folio);
struct f2fs_node *rn = CACHED_NODE(entry);
struct node_footer *footer = F2FS_NODE_FOOTER(sbi, entry);
unsigned int old_flag = 0;
if (reset)
memset(rn, 0, sizeof(*rn));
memset(rn, 0, F2FS_BLKSIZE(sbi));
else
old_flag = le32_to_cpu(rn->footer.flag);
old_flag = le32_to_cpu(footer->flag);
rn->footer.nid = cpu_to_le32(nid);
rn->footer.ino = cpu_to_le32(ino);
memset(footer, 0, sizeof(*footer));
footer->nid = cpu_to_le32(nid);
footer->ino = cpu_to_le32(ino);
/* should remain old flag bits such as COLD_BIT_SHIFT */
rn->footer.flag = cpu_to_le32((ofs << OFFSET_BIT_SHIFT) |
footer->flag = cpu_to_le32((ofs << OFFSET_BIT_SHIFT) |
(old_flag & OFFSET_BIT_MASK));
}
static inline void copy_node_footer(const struct folio *dst,
const struct folio *src)
static inline void copy_node_footer(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *dst,
const struct f2fs_cached_block *src)
{
struct f2fs_node *src_rn = F2FS_NODE(src);
struct f2fs_node *dst_rn = F2FS_NODE(dst);
memcpy(&dst_rn->footer, &src_rn->footer, sizeof(struct node_footer));
memcpy(F2FS_NODE_FOOTER(sbi, dst), F2FS_NODE_FOOTER(sbi, src),
sizeof(struct node_footer));
}
static inline void fill_node_footer_blkaddr(struct folio *folio, block_t blkaddr)
static inline void fill_node_footer_blkaddr(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, block_t blkaddr)
{
struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio));
struct f2fs_node *rn = F2FS_NODE(folio);
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
struct node_footer *footer = F2FS_NODE_FOOTER(sbi, entry);
__u64 cp_ver = cur_cp_version(ckpt);
if (__is_set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG))
cp_ver |= (cur_cp_crc(ckpt) << 32);
rn->footer.cp_ver = cpu_to_le64(cp_ver);
rn->footer.next_blkaddr = cpu_to_le32(blkaddr);
footer->cp_ver = cpu_to_le64(cp_ver);
footer->next_blkaddr = cpu_to_le32(blkaddr);
}
static inline bool is_recoverable_dnode(const struct folio *folio)
static inline bool is_recoverable_dnode(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
struct f2fs_checkpoint *ckpt = F2FS_CKPT(F2FS_F_SB(folio));
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
__u64 cp_ver = cur_cp_version(ckpt);
/* Don't care crc part, if fsck.f2fs sets it. */
if (__is_set_ckpt_flags(ckpt, CP_NOCRC_RECOVERY_FLAG))
return (cp_ver << 32) == (cpver_of_node(folio) << 32);
return (cp_ver << 32) == (cpver_of_node(sbi, entry) << 32);
if (__is_set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG))
cp_ver |= (cur_cp_crc(ckpt) << 32);
return cp_ver == cpver_of_node(folio);
return cp_ver == cpver_of_node(sbi, entry);
}
/*
@ -347,43 +362,50 @@ static inline bool is_recoverable_dnode(const struct folio *folio)
* `- indirect node ((6 + 2N) + (N - 1)(N + 1))
* `- direct node
*/
static inline bool IS_DNODE(const struct folio *node_folio)
static inline bool IS_DNODE(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry)
{
unsigned int ofs = ofs_of_node(node_folio);
unsigned int ofs = ofs_of_node(sbi, entry);
if (f2fs_has_xattr_block(ofs))
return true;
if (ofs == 3 || ofs == 4 + NIDS_PER_BLOCK ||
ofs == 5 + 2 * NIDS_PER_BLOCK)
if (ofs == 3 || ofs == 4 + NIDS_PER_BLOCK(sbi) ||
ofs == 5 + 2 * NIDS_PER_BLOCK(sbi))
return false;
if (ofs >= 6 + 2 * NIDS_PER_BLOCK) {
ofs -= 6 + 2 * NIDS_PER_BLOCK;
if (!((long int)ofs % (NIDS_PER_BLOCK + 1)))
if (ofs >= 6 + 2 * NIDS_PER_BLOCK(sbi)) {
ofs -= 6 + 2 * NIDS_PER_BLOCK(sbi);
if (!((long)ofs % (NIDS_PER_BLOCK(sbi) + 1)))
return false;
}
return true;
}
static inline int set_nid(struct folio *folio, int off, nid_t nid, bool i)
static inline bool set_nid(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, int off, nid_t nid, bool i)
{
struct f2fs_node *rn = F2FS_NODE(folio);
struct f2fs_node *rn = CACHED_NODE(entry);
__le32 *inode_nids = F2FS_INODE_NIDS(sbi, entry);
__le32 *addr = i ? &inode_nids[off - NODE_DIR1_BLOCK(sbi)] : &rn->in.nid[off];
f2fs_folio_wait_writeback(folio, NODE, true, true);
f2fs_cache_wait_writeback(entry);
if (*addr == cpu_to_le32(nid))
return false;
if (i)
rn->i.i_nid[off - NODE_DIR1_BLOCK] = cpu_to_le32(nid);
else
rn->in.nid[off] = cpu_to_le32(nid);
return folio_mark_dirty(folio);
*addr = cpu_to_le32(nid);
f2fs_mark_cache_dirty(entry);
return true;
}
static inline nid_t get_nid(const struct folio *folio, int off, bool i)
static inline nid_t get_nid(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry, int off, bool i)
{
struct f2fs_node *rn = F2FS_NODE(folio);
struct f2fs_node *rn = CACHED_NODE(entry);
const __le32 *inode_nids = F2FS_INODE_NIDS(sbi, entry);
int nid_index = off - NODE_DIR1_BLOCK(sbi);
if (i)
return le32_to_cpu(rn->i.i_nid[off - NODE_DIR1_BLOCK]);
return le32_to_cpu(inode_nids[nid_index]);
return le32_to_cpu(rn->in.nid[off]);
}
@ -394,40 +416,43 @@ static inline nid_t get_nid(const struct folio *folio, int off, bool i)
* - Mark cold data pages in page cache
*/
static inline int is_node(const struct folio *folio, int type)
static inline int is_node(struct f2fs_sb_info *sbi,
const struct f2fs_cached_block *entry, int type)
{
struct f2fs_node *rn = F2FS_NODE(folio);
return le32_to_cpu(rn->footer.flag) & BIT(type);
return le32_to_cpu(F2FS_NODE_FOOTER(sbi, entry)->flag) & BIT(type);
}
#define is_cold_node(folio) is_node(folio, COLD_BIT_SHIFT)
#define is_fsync_dnode(folio) is_node(folio, FSYNC_BIT_SHIFT)
#define is_dent_dnode(folio) is_node(folio, DENT_BIT_SHIFT)
#define is_cold_node(sbi, entry) is_node(sbi, entry, COLD_BIT_SHIFT)
#define is_fsync_dnode(sbi, entry) is_node(sbi, entry, FSYNC_BIT_SHIFT)
#define is_dent_dnode(sbi, entry) is_node(sbi, entry, DENT_BIT_SHIFT)
static inline void __set_mark(const struct folio *folio, bool mark, int type)
static inline void __set_mark(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, bool mark, int type)
{
struct f2fs_node *rn = F2FS_NODE(folio);
unsigned int flag = le32_to_cpu(rn->footer.flag);
struct node_footer *footer = F2FS_NODE_FOOTER(sbi, entry);
unsigned int flag = le32_to_cpu(footer->flag);
if (mark)
flag |= BIT(type);
else
flag &= ~BIT(type);
rn->footer.flag = cpu_to_le32(flag);
footer->flag = cpu_to_le32(flag);
}
static inline void set_cold_node(const struct folio *folio, bool is_dir)
static inline void set_cold_node(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, bool is_dir)
{
__set_mark(folio, !is_dir, COLD_BIT_SHIFT);
__set_mark(sbi, entry, !is_dir, COLD_BIT_SHIFT);
}
static inline void set_mark(struct folio *folio, bool mark, int type)
static inline void set_mark(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, bool mark, int type)
{
__set_mark(folio, mark, type);
__set_mark(sbi, entry, mark, type);
#ifdef CONFIG_F2FS_CHECK_FS
f2fs_inode_chksum_set(F2FS_F_SB(folio), folio);
f2fs_inode_chksum_set(sbi, entry);
#endif
}
#define set_dentry_mark(folio, mark) set_mark(folio, mark, DENT_BIT_SHIFT)
#define set_fsync_mark(folio, mark) set_mark(folio, mark, FSYNC_BIT_SHIFT)
#define set_dentry_mark(sbi, entry, mark) set_mark(sbi, entry, mark, DENT_BIT_SHIFT)
#define set_fsync_mark(sbi, entry, mark) set_mark(sbi, entry, mark, FSYNC_BIT_SHIFT)

255
fs/f2fs/recovery.c
View File

@ -182,38 +182,39 @@ static const char *recover_printable_name(struct inode *inode,
return raw->i_name;
}
static int recover_dentry(struct inode *inode, struct folio *ifolio,
static int recover_dentry(struct inode *inode, struct f2fs_cached_block *entry,
struct list_head *dir_list)
{
struct f2fs_inode *raw_inode = F2FS_INODE(ifolio);
struct f2fs_inode *raw_inode = &CACHED_NODE(entry)->i;
nid_t pino = le32_to_cpu(raw_inode->i_pino);
struct f2fs_dir_entry *de;
struct f2fs_filename fname;
struct qstr usr_fname;
struct folio *folio;
void *dentry_blk = NULL;
struct inode *dir, *einode;
struct fsync_inode_entry *entry;
struct fsync_inode_entry *fsync_entry;
int err = 0;
const char *name;
int name_len;
entry = get_fsync_inode(dir_list, pino);
if (!entry) {
entry = add_fsync_inode(F2FS_I_SB(inode), dir_list,
fsync_entry = get_fsync_inode(dir_list, pino);
if (!fsync_entry) {
fsync_entry = add_fsync_inode(F2FS_I_SB(inode), dir_list,
pino, false);
if (IS_ERR(entry)) {
dir = ERR_CAST(entry);
err = PTR_ERR(entry);
if (IS_ERR(fsync_entry)) {
dir = ERR_CAST(fsync_entry);
err = PTR_ERR(fsync_entry);
goto out;
}
}
dir = entry->inode;
dir = fsync_entry->inode;
err = init_recovered_filename(dir, inode, raw_inode, &fname, &usr_fname);
if (err)
goto out;
retry:
de = __f2fs_find_entry(dir, &fname, &folio);
dentry_blk = NULL;
de = __f2fs_find_entry(dir, &fname, &dentry_blk);
if (de && inode->i_ino == le32_to_cpu(de->ino))
goto out_put;
@ -238,11 +239,11 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio,
iput(einode);
goto out_put;
}
f2fs_delete_entry(de, folio, dir, einode);
f2fs_delete_entry(de, dentry_blk, dir, einode);
iput(einode);
goto retry;
} else if (IS_ERR(folio)) {
err = PTR_ERR(folio);
} else if (IS_ERR(dentry_blk)) {
err = PTR_ERR(dentry_blk);
} else {
err = f2fs_add_dentry(dir, &fname, inode,
inode->i_ino, inode->i_mode);
@ -252,18 +253,18 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio,
goto out;
out_put:
f2fs_folio_put(folio, false);
f2fs_put_dentry_block(dentry_blk, false);
out:
name = recover_printable_name(inode, raw_inode, &name_len);
f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d",
__func__, ino_of_node(ifolio), name_len, name,
__func__, ino_of_node(F2FS_I_SB(inode), entry), name_len, name,
IS_ERR(dir) ? 0 : dir->i_ino, err);
return err;
}
static int recover_quota_data(struct inode *inode, struct folio *folio)
static int recover_quota_data(struct inode *inode, struct f2fs_cached_block *entry)
{
struct f2fs_inode *raw = F2FS_INODE(folio);
struct f2fs_inode *raw = &CACHED_NODE(entry)->i;
struct iattr attr;
uid_t i_uid = le32_to_cpu(raw->i_uid);
gid_t i_gid = le32_to_cpu(raw->i_gid);
@ -300,9 +301,9 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri)
clear_inode_flag(inode, FI_DATA_EXIST);
}
static int recover_inode(struct inode *inode, struct folio *folio)
static int recover_inode(struct inode *inode, struct f2fs_cached_block *entry)
{
struct f2fs_inode *raw = F2FS_INODE(folio);
struct f2fs_inode *raw = &CACHED_NODE(entry)->i;
struct f2fs_inode_info *fi = F2FS_I(inode);
const char *name;
int name_len;
@ -310,7 +311,7 @@ static int recover_inode(struct inode *inode, struct folio *folio)
inode->i_mode = le16_to_cpu(raw->i_mode);
err = recover_quota_data(inode, folio);
err = recover_quota_data(inode, entry);
if (err)
return err;
@ -357,7 +358,7 @@ static int recover_inode(struct inode *inode, struct folio *folio)
name = recover_printable_name(inode, raw, &name_len);
f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x",
__func__, ino_of_node(folio), name_len, name,
__func__, ino_of_node(F2FS_I_SB(inode), entry), name_len, name,
raw->i_inline);
return 0;
}
@ -386,30 +387,30 @@ static int sanity_check_node_chain(struct f2fs_sb_info *sbi, block_t blkaddr,
return 0;
for (i = 0; i < 2; i++) {
struct folio *folio;
struct f2fs_cached_block *entry;
if (!f2fs_is_valid_blkaddr(sbi, *blkaddr_fast, META_POR)) {
*is_detecting = false;
return 0;
}
folio = f2fs_get_tmp_folio(sbi, *blkaddr_fast);
if (IS_ERR(folio))
return PTR_ERR(folio);
entry = f2fs_get_tmp_cache(sbi, *blkaddr_fast);
if (IS_ERR(entry))
return PTR_ERR(entry);
if (!is_recoverable_dnode(folio)) {
f2fs_folio_put(folio, true);
if (!is_recoverable_dnode(sbi, entry)) {
f2fs_put_cache(entry, true);
*is_detecting = false;
return 0;
}
ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, *blkaddr_fast,
next_blkaddr_of_node(folio));
next_blkaddr_of_node(sbi, entry));
*blkaddr_fast = next_blkaddr_of_node(folio);
f2fs_folio_put(folio, true);
*blkaddr_fast = next_blkaddr_of_node(sbi, entry);
f2fs_put_cache(entry, true);
f2fs_ra_meta_pages_cond(sbi, *blkaddr_fast, ra_blocks);
f2fs_ra_meta_caches_cond(sbi, *blkaddr_fast, ra_blocks);
}
if (*blkaddr_fast == blkaddr) {
@ -434,45 +435,45 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
blkaddr_fast = blkaddr;
while (1) {
struct fsync_inode_entry *entry;
struct folio *folio;
struct fsync_inode_entry *fsync_entry;
struct f2fs_cached_block *entry;
if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR))
return 0;
folio = f2fs_get_tmp_folio(sbi, blkaddr);
if (IS_ERR(folio)) {
err = PTR_ERR(folio);
entry = f2fs_get_tmp_cache(sbi, blkaddr);
if (IS_ERR(entry)) {
err = PTR_ERR(entry);
break;
}
if (!is_recoverable_dnode(folio)) {
f2fs_folio_put(folio, true);
if (!is_recoverable_dnode(sbi, entry)) {
f2fs_put_cache(entry, true);
break;
}
if (!is_fsync_dnode(folio))
if (!is_fsync_dnode(sbi, entry))
goto next;
entry = get_fsync_inode(head, ino_of_node(folio));
if (!entry) {
fsync_entry = get_fsync_inode(head, ino_of_node(sbi, entry));
if (!fsync_entry) {
bool quota_inode = false;
if (!check_only &&
IS_INODE(folio) &&
is_dent_dnode(folio)) {
err = f2fs_recover_inode_page(sbi, folio);
IS_INODE(sbi, entry) &&
is_dent_dnode(sbi, entry)) {
err = f2fs_recover_inode_cache(sbi, entry);
if (err) {
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
break;
}
quota_inode = true;
}
entry = add_fsync_inode(sbi, head, ino_of_node(folio),
fsync_entry = add_fsync_inode(sbi, head, ino_of_node(sbi, entry),
quota_inode);
if (IS_ERR(entry)) {
err = PTR_ERR(entry);
if (IS_ERR(fsync_entry)) {
err = PTR_ERR(fsync_entry);
/*
* CP | dnode(F) | inode(DF)
* For this case, we should not give up now.
@ -482,18 +483,18 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
*new_inode = true;
goto next;
}
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
break;
}
}
entry->blkaddr = blkaddr;
fsync_entry->blkaddr = blkaddr;
if (IS_INODE(folio) && is_dent_dnode(folio))
entry->last_dentry = blkaddr;
if (IS_INODE(sbi, entry) && is_dent_dnode(sbi, entry))
fsync_entry->last_dentry = blkaddr;
next:
/* check next segment */
blkaddr = next_blkaddr_of_node(folio);
f2fs_folio_put(folio, true);
blkaddr = next_blkaddr_of_node(sbi, entry);
f2fs_put_cache(entry, true);
err = sanity_check_node_chain(sbi, blkaddr, &blkaddr_fast,
&is_detecting);
@ -519,7 +520,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
unsigned short blkoff = GET_BLKOFF_FROM_SEG0(sbi, blkaddr);
struct f2fs_summary_block *sum_node;
struct f2fs_summary sum;
struct folio *sum_folio, *node_folio;
struct f2fs_cached_block *entry = NULL;
struct f2fs_cached_block *node_entry;
struct dnode_of_data tdn = *dn;
nid_t ino, nid;
struct inode *inode;
@ -541,18 +543,18 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
}
}
sum_folio = f2fs_get_sum_folio(sbi, segno);
if (IS_ERR(sum_folio))
return PTR_ERR(sum_folio);
sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, segno);
entry = f2fs_get_sum_cache(sbi, segno);
if (IS_ERR(entry))
return PTR_ERR(entry);
sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, segno);
sum = sum_entries(sum_node)[blkoff];
f2fs_folio_put(sum_folio, true);
f2fs_put_cache(entry, true);
got_it:
/* Use the locked dnode page and inode */
nid = le32_to_cpu(sum.nid);
ofs_in_node = le16_to_cpu(sum.ofs_in_node);
max_addrs = ADDRS_PER_PAGE(dn->node_folio, dn->inode);
max_addrs = ADDRS_PER_PAGE(dn->node_entry, dn->inode);
if (ofs_in_node >= max_addrs) {
f2fs_err(sbi, "Inconsistent ofs_in_node:%u in summary, ino:%llu, nid:%u, max:%u",
ofs_in_node, dn->inode->i_ino, nid, max_addrs);
@ -562,9 +564,9 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
if (dn->inode->i_ino == nid) {
tdn.nid = nid;
if (!dn->inode_folio_locked)
folio_lock(dn->inode_folio);
tdn.node_folio = dn->inode_folio;
if (!dn->inode_entry_locked)
f2fs_lock_cache(dn->inode_entry);
tdn.node_entry = dn->inode_entry;
tdn.ofs_in_node = ofs_in_node;
goto truncate_out;
} else if (dn->nid == nid) {
@ -573,13 +575,13 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
}
/* Get the node page */
node_folio = f2fs_get_node_folio(sbi, nid, NODE_TYPE_REGULAR);
if (IS_ERR(node_folio))
return PTR_ERR(node_folio);
node_entry = f2fs_get_node_cache(sbi, nid, NODE_TYPE_REGULAR);
if (IS_ERR(node_entry))
return PTR_ERR(node_entry);
offset = ofs_of_node(node_folio);
ino = ino_of_node(node_folio);
f2fs_folio_put(node_folio, true);
offset = ofs_of_node(sbi, node_entry);
ino = ino_of_node(sbi, node_entry);
f2fs_put_cache(node_entry, true);
if (ino != dn->inode->i_ino) {
int ret;
@ -605,8 +607,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
* if inode page is locked, unlock temporarily, but its reference
* count keeps alive.
*/
if (ino == dn->inode->i_ino && dn->inode_folio_locked)
folio_unlock(dn->inode_folio);
if (ino == dn->inode->i_ino && dn->inode_entry_locked)
f2fs_unlock_cache(dn->inode_entry);
set_new_dnode(&tdn, inode, NULL, NULL, 0);
if (f2fs_get_dnode_of_data(&tdn, bidx, LOOKUP_NODE))
@ -619,15 +621,15 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
out:
if (ino != dn->inode->i_ino)
iput(inode);
else if (dn->inode_folio_locked)
folio_lock(dn->inode_folio);
else if (dn->inode_entry_locked)
f2fs_lock_cache(dn->inode_entry);
return 0;
truncate_out:
if (f2fs_data_blkaddr(&tdn) == blkaddr)
f2fs_truncate_data_blocks_range(&tdn, 1);
if (dn->inode->i_ino == nid && !dn->inode_folio_locked)
folio_unlock(dn->inode_folio);
if (dn->inode->i_ino == nid && !dn->inode_entry_locked)
f2fs_unlock_cache(dn->inode_entry);
return 0;
}
@ -645,7 +647,7 @@ static int f2fs_reserve_new_block_retry(struct dnode_of_data *dn)
}
static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
struct folio *folio)
struct f2fs_cached_block *entry)
{
struct dnode_of_data dn;
struct node_info ni;
@ -653,19 +655,19 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
int err = 0, recovered = 0;
/* step 1: recover xattr */
if (IS_INODE(folio)) {
err = f2fs_recover_inline_xattr(inode, folio);
if (IS_INODE(sbi, entry)) {
err = f2fs_recover_inline_xattr(inode, entry);
if (err)
goto out;
} else if (f2fs_has_xattr_block(ofs_of_node(folio))) {
err = f2fs_recover_xattr_data(inode, folio);
} else if (f2fs_has_xattr_block(ofs_of_node(sbi, entry))) {
err = f2fs_recover_xattr_data(inode, entry);
if (!err)
recovered++;
goto out;
}
/* step 2: recover inline data */
err = f2fs_recover_inline_data(inode, folio);
err = f2fs_recover_inline_data(inode, entry);
if (err) {
if (err == 1)
err = 0;
@ -673,8 +675,8 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
}
/* step 3: recover data indices */
start = f2fs_start_bidx_of_node(ofs_of_node(folio), inode);
end = start + ADDRS_PER_PAGE(folio, inode);
start = f2fs_start_bidx_of_node(ofs_of_node(sbi, entry), inode);
end = start + addrs_per_page(inode, IS_INODE(sbi, entry));
set_new_dnode(&dn, inode, NULL, NULL, 0);
retry_dn:
@ -687,18 +689,18 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
goto out;
}
f2fs_folio_wait_writeback(dn.node_folio, NODE, true, true);
f2fs_cache_wait_writeback(dn.node_entry);
err = f2fs_get_node_info(sbi, dn.nid, &ni, false);
if (err)
goto err;
f2fs_bug_on(sbi, ni.ino != ino_of_node(folio));
f2fs_bug_on(sbi, ni.ino != ino_of_node(sbi, entry));
if (ofs_of_node(dn.node_folio) != ofs_of_node(folio)) {
if (ofs_of_node(sbi, dn.node_entry) != ofs_of_node(sbi, entry)) {
f2fs_warn(sbi, "Inconsistent ofs_of_node, ino:%llu, ofs:%u, %u",
inode->i_ino, ofs_of_node(dn.node_folio),
ofs_of_node(folio));
inode->i_ino, ofs_of_node(sbi, dn.node_entry),
ofs_of_node(sbi, entry));
err = -EFSCORRUPTED;
f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER);
fserror_report_file_metadata(dn.inode, err, GFP_NOFS);
@ -709,7 +711,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
block_t src, dest;
src = f2fs_data_blkaddr(&dn);
dest = data_blkaddr(dn.inode, folio, dn.ofs_in_node);
dest = data_blkaddr(dn.inode, entry, dn.ofs_in_node);
if (__is_valid_data_blkaddr(src) &&
!f2fs_is_valid_blkaddr(sbi, src, META_POR)) {
@ -734,9 +736,9 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
}
if (!file_keep_isize(inode) &&
(i_size_read(inode) <= ((loff_t)index << PAGE_SHIFT)))
(i_size_read(inode) <= F2FS_BLK_TO_BYTES(sbi, index)))
f2fs_i_size_write(inode,
(loff_t)(index + 1) << PAGE_SHIFT);
F2FS_BLK_TO_BYTES(sbi, index + 1));
/*
* dest is reserved block, invalidate src block
@ -784,16 +786,16 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
}
}
copy_node_footer(dn.node_folio, folio);
fill_node_footer(dn.node_folio, dn.nid, ni.ino,
ofs_of_node(folio), false);
folio_mark_dirty(dn.node_folio);
copy_node_footer(sbi, dn.node_entry, entry);
fill_node_footer(sbi, dn.node_entry, dn.nid, ni.ino,
ofs_of_node(sbi, entry), false);
f2fs_mark_cache_dirty(dn.node_entry);
err:
f2fs_put_dnode(&dn);
out:
f2fs_notice(sbi, "recover_data: ino = %llx, nid = %x (i_size: %s), "
"range (%u, %u), recovered = %d, err = %d",
inode->i_ino, nid_of_node(folio),
inode->i_ino, nid_of_node(sbi, entry),
file_keep_isize(inode) ? "keep" : "recover",
start, end, recovered, err);
return err;
@ -820,26 +822,26 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list,
blkaddr = NEXT_FREE_BLKADDR(sbi, curseg);
while (1) {
struct fsync_inode_entry *entry;
struct folio *folio;
struct fsync_inode_entry *fsync_entry;
struct f2fs_cached_block *entry;
if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR))
break;
folio = f2fs_get_tmp_folio(sbi, blkaddr);
if (IS_ERR(folio)) {
err = PTR_ERR(folio);
entry = f2fs_get_tmp_cache(sbi, blkaddr);
if (IS_ERR(entry)) {
err = PTR_ERR(entry);
break;
}
if (!is_recoverable_dnode(folio)) {
f2fs_folio_put(folio, true);
if (!is_recoverable_dnode(sbi, entry)) {
f2fs_put_cache(entry, true);
break;
}
recoverable_dnode++;
entry = get_fsync_inode(inode_list, ino_of_node(folio));
if (!entry)
fsync_entry = get_fsync_inode(inode_list, ino_of_node(sbi, entry));
if (!fsync_entry)
goto next;
fsynced_dnode++;
/*
@ -847,40 +849,40 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list,
* In this case, we can lose the latest inode(x).
* So, call recover_inode for the inode update.
*/
if (IS_INODE(folio)) {
err = recover_inode(entry->inode, folio);
if (IS_INODE(sbi, entry)) {
err = recover_inode(fsync_entry->inode, entry);
if (err) {
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
break;
}
recovered_inode++;
}
if (entry->last_dentry == blkaddr) {
err = recover_dentry(entry->inode, folio, dir_list);
if (fsync_entry->last_dentry == blkaddr) {
err = recover_dentry(fsync_entry->inode, entry, dir_list);
if (err) {
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
break;
}
recovered_dentry++;
}
err = do_recover_data(sbi, entry->inode, folio);
err = do_recover_data(sbi, fsync_entry->inode, entry);
if (err) {
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
break;
}
recovered_dnode++;
if (entry->blkaddr == blkaddr)
list_move_tail(&entry->list, tmp_inode_list);
if (fsync_entry->blkaddr == blkaddr)
list_move_tail(&fsync_entry->list, tmp_inode_list);
next:
ra_blocks = adjust_por_ra_blocks(sbi, ra_blocks, blkaddr,
next_blkaddr_of_node(folio));
next_blkaddr_of_node(sbi, entry));
/* check next segment */
blkaddr = next_blkaddr_of_node(folio);
f2fs_folio_put(folio, true);
blkaddr = next_blkaddr_of_node(sbi, entry);
f2fs_put_cache(entry, true);
f2fs_ra_meta_pages_cond(sbi, blkaddr, ra_blocks);
f2fs_ra_meta_caches_cond(sbi, blkaddr, ra_blocks);
total_dnode++;
}
if (!err)
@ -937,12 +939,11 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
destroy_fsync_dnodes(&tmp_inode_list, err);
/* truncate meta pages to be used by the recovery */
truncate_inode_pages_range(META_MAPPING(sbi),
(loff_t)MAIN_BLKADDR(sbi) << PAGE_SHIFT, -1);
f2fs_truncate_meta_caches(sbi, MAIN_BLKADDR(sbi),
MAX_BLKADDR(sbi) - MAIN_BLKADDR(sbi));
if (err) {
truncate_inode_pages_final(NODE_MAPPING(sbi));
truncate_inode_pages_final(META_MAPPING(sbi));
f2fs_truncate_node_caches(sbi, 0, ULONG_MAX);
f2fs_truncate_meta_caches(sbi, 0, ULONG_MAX);
}
/*

447
fs/f2fs/segment.c
View File

@ -194,7 +194,7 @@ void f2fs_abort_atomic_write(struct inode *inode, bool clean)
return;
if (clean)
truncate_inode_pages_final(inode->i_mapping);
truncate_inode_pages(inode->i_mapping, 0);
release_atomic_write_cnt(inode);
clear_inode_flag(inode, FI_ATOMIC_COMMITTED);
@ -335,8 +335,8 @@ static int __f2fs_commit_atomic_write(struct inode *inode)
goto next;
}
blen = min((pgoff_t)ADDRS_PER_PAGE(dn.node_folio, cow_inode),
len);
blen = min((pgoff_t)addrs_per_page(cow_inode,
IS_INODE(F2FS_I_SB(cow_inode), dn.node_entry)), len);
index = off;
for (i = 0; i < blen; i++, dn.ofs_in_node++, index++) {
blkaddr = f2fs_data_blkaddr(&dn);
@ -479,11 +479,11 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need)
static inline bool excess_dirty_threshold(struct f2fs_sb_info *sbi)
{
int factor = f2fs_rwsem_is_locked(&sbi->cp_rwsem) ? 3 : 2;
unsigned int dents = get_pages(sbi, F2FS_DIRTY_DENTS);
unsigned int qdata = get_pages(sbi, F2FS_DIRTY_QDATA);
unsigned int nodes = get_pages(sbi, F2FS_DIRTY_NODES);
unsigned int meta = get_pages(sbi, F2FS_DIRTY_META);
unsigned int imeta = get_pages(sbi, F2FS_DIRTY_IMETA);
unsigned int dents = get_nr_caches(sbi, F2FS_DIRTY_DENTS);
unsigned int qdata = get_nr_caches(sbi, F2FS_DIRTY_QDATA);
unsigned int nodes = get_nr_caches(sbi, F2FS_DIRTY_NODES);
unsigned int meta = get_nr_caches(sbi, F2FS_DIRTY_META);
unsigned int imeta = get_nr_caches(sbi, F2FS_DIRTY_IMETA);
unsigned int threshold =
SEGS_TO_BLKS(sbi, (factor * DEFAULT_DIRTY_THRESHOLD));
unsigned int global_threshold = threshold * 3 / 2;
@ -510,12 +510,20 @@ void f2fs_balance_fs_bg(struct f2fs_sb_info *sbi, bool from_bg)
f2fs_shrink_age_extent_tree(sbi,
AGE_EXTENT_CACHE_SHRINK_NUMBER);
/* try to shrink meta cache when there is no enough memory */
if (!f2fs_available_free_memory(sbi, META_BLOCK))
f2fs_shrink_meta_cache(sbi, METADATA_CACHE_SHRINK_NUMBER);
/* try to shrink node cache when there is no enough memory */
if (!f2fs_available_free_memory(sbi, NODE_BLOCK))
f2fs_shrink_node_cache(sbi, METADATA_CACHE_SHRINK_NUMBER);
/* check the # of cached NAT entries */
if (!f2fs_available_free_memory(sbi, NAT_ENTRIES))
f2fs_try_to_free_nats(sbi, NAT_ENTRY_PER_BLOCK);
f2fs_try_to_free_nats(sbi, NAT_ENTRY_PER_BLOCK(sbi));
if (!f2fs_available_free_memory(sbi, FREE_NIDS))
f2fs_try_to_free_nids(sbi, MAX_FREE_NIDS);
f2fs_try_to_free_nids(sbi, MAX_FREE_NIDS(sbi));
else
f2fs_build_free_nids(sbi, false, false);
@ -1310,13 +1318,14 @@ static void __submit_zone_reset_cmd(struct f2fs_sb_info *sbi,
/* sanity check on discard range */
__check_sit_bitmap(sbi, dc->di.lstart, dc->di.lstart + dc->di.len);
bio->bi_iter.bi_sector = SECTOR_FROM_BLOCK(dc->di.start);
bio->bi_iter.bi_sector = SECTOR_FROM_BLOCK(sbi, dc->di.start);
bio->bi_private = dc;
bio->bi_end_io = f2fs_submit_discard_endio;
submit_bio(bio);
atomic_inc(&dcc->issued_discard);
f2fs_update_iostat(sbi, NULL, FS_ZONE_RESET_IO, dc->di.len * F2FS_BLKSIZE);
f2fs_update_iostat(sbi, NULL, FS_ZONE_RESET_IO,
dc->di.len * F2FS_BLKSIZE(sbi));
}
#endif
@ -1327,7 +1336,7 @@ static int __submit_discard_cmd(struct f2fs_sb_info *sbi,
{
struct block_device *bdev = dc->bdev;
unsigned int max_discard_blocks =
SECTOR_TO_BLOCK(bdev_max_discard_sectors(bdev));
SECTOR_TO_BLOCK(sbi, bdev_max_discard_sectors(bdev));
struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
struct list_head *wait_list = (dpolicy->type == DPOLICY_FSTRIM) ?
&(dcc->fstrim_list) : &(dcc->wait_list);
@ -1389,8 +1398,8 @@ static int __submit_discard_cmd(struct f2fs_sb_info *sbi,
dc->di.len += len;
__blkdev_issue_discard(bdev, SECTOR_FROM_BLOCK(start),
SECTOR_FROM_BLOCK(len), GFP_NOFS, &bio);
__blkdev_issue_discard(bdev, SECTOR_FROM_BLOCK(sbi, start),
SECTOR_FROM_BLOCK(sbi, len), GFP_NOFS, &bio);
f2fs_bug_on(sbi, !bio);
/*
@ -1419,7 +1428,8 @@ static int __submit_discard_cmd(struct f2fs_sb_info *sbi,
atomic_inc(&dcc->issued_discard);
f2fs_update_iostat(sbi, NULL, FS_DISCARD_IO, len * F2FS_BLKSIZE);
f2fs_update_iostat(sbi, NULL, FS_DISCARD_IO,
len * F2FS_BLKSIZE(sbi));
lstart += len;
start += len;
@ -1518,7 +1528,7 @@ static void __update_discard_tree_range(struct f2fs_sb_info *sbi,
struct discard_info di = {0};
struct rb_node **insert_p = NULL, *insert_parent = NULL;
unsigned int max_discard_blocks =
SECTOR_TO_BLOCK(bdev_max_discard_sectors(bdev));
SECTOR_TO_BLOCK(sbi, bdev_max_discard_sectors(bdev));
block_t end = lstart + len;
dc = __lookup_discard_cmd_ret(&dcc->root, lstart,
@ -1871,6 +1881,60 @@ static unsigned int __wait_all_discard_cmd(struct f2fs_sb_info *sbi,
return discard_blks;
}
void f2fs_drop_discard_cmd_range(struct f2fs_sb_info *sbi,
block_t start, block_t len)
{
struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
struct discard_cmd *prev_dc = NULL, *next_dc = NULL;
struct rb_node **insert_p = NULL, *insert_parent = NULL;
struct discard_cmd *dc, *wait_dc;
u64 cur = start;
u64 end = (u64)start + len;
int count;
if (!f2fs_realtime_discard_enable(sbi))
return;
next:
count = 0;
wait_dc = NULL;
mutex_lock(&dcc->cmd_lock);
while (cur < end) {
dc = __lookup_discard_cmd_ret(&dcc->root, cur,
&prev_dc, &next_dc, &insert_p, &insert_parent);
if (!dc)
dc = next_dc;
if (!dc || (u64)dc->di.lstart >= end)
break;
if (dc->state == D_PREP) {
cur = (u64)dc->di.lstart + dc->di.len;
__remove_discard_cmd(sbi, dc);
if (++count >= MAX_DISCARD_DROP_COUNT)
break;
continue;
}
dc->ref++;
cur = (u64)dc->di.lstart + dc->di.len;
wait_dc = dc;
break;
}
mutex_unlock(&dcc->cmd_lock);
if (wait_dc) {
__wait_one_discard_bio(sbi, wait_dc);
goto next;
}
if (count >= MAX_DISCARD_DROP_COUNT) {
cond_resched();
goto next;
}
}
/* This should be covered by global mutex, &sit_i->sentry_lock */
static void f2fs_wait_discard_bio(struct f2fs_sb_info *sbi, block_t blkaddr)
{
@ -2041,8 +2105,8 @@ static int __f2fs_issue_discard_zone(struct f2fs_sb_info *sbi,
/* For sequential zones, reset the zone write pointer */
if (f2fs_blkz_is_seq(sbi, devi, blkstart)) {
sector = SECTOR_FROM_BLOCK(blkstart);
nr_sects = SECTOR_FROM_BLOCK(blklen);
sector = SECTOR_FROM_BLOCK(sbi, blkstart);
nr_sects = SECTOR_FROM_BLOCK(sbi, blklen);
div64_u64_rem(sector, bdev_zone_sectors(bdev), &remainder);
if (remainder || nr_sects != bdev_zone_sectors(bdev)) {
@ -2749,12 +2813,12 @@ static unsigned short f2fs_curseg_valid_blocks(struct f2fs_sb_info *sbi, int typ
}
/*
* Calculate the number of current summary pages for writing
* Calculate the number of current summary blocks for writing
*/
int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra)
int f2fs_nblocks_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra)
{
int valid_sum_count = 0;
int i, sum_in_page;
int i, sum_in_block;
for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_DATA; i++) {
if (sbi->ckpt->alloc_type[i] != SSR && for_ra)
@ -2764,73 +2828,66 @@ int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra)
valid_sum_count += f2fs_curseg_valid_blocks(sbi, i);
}
sum_in_page = (sbi->blocksize - 2 * sbi->sum_journal_size -
sum_in_block = (sbi->blocksize - 2 * sbi->sum_journal_size -
SUM_FOOTER_SIZE) / SUMMARY_SIZE;
if (valid_sum_count <= sum_in_page)
if (valid_sum_count <= sum_in_block)
return 1;
else if ((valid_sum_count - sum_in_page) <=
else if ((valid_sum_count - sum_in_block) <=
(sbi->blocksize - SUM_FOOTER_SIZE) / SUMMARY_SIZE)
return 2;
return 3;
}
/*
* Caller should put this summary folio
*/
struct folio *f2fs_get_sum_folio(struct f2fs_sb_info *sbi, unsigned int segno)
struct f2fs_cached_block *f2fs_get_sum_cache(struct f2fs_sb_info *sbi,
unsigned int segno)
{
if (unlikely(f2fs_cp_error(sbi)))
return ERR_PTR(-EIO);
return f2fs_get_meta_folio_retry(sbi, GET_SUM_BLOCK(sbi, segno));
return f2fs_get_meta_cache_retry(sbi, GET_SUM_BLOCK(sbi, segno));
}
void f2fs_update_meta_page(struct f2fs_sb_info *sbi,
void f2fs_update_meta_block(struct f2fs_sb_info *sbi,
void *src, block_t blk_addr)
{
struct folio *folio;
struct f2fs_cached_block *entry;
if (!f2fs_sb_has_packed_ssa(sbi))
folio = f2fs_grab_meta_folio(sbi, blk_addr);
else
folio = f2fs_get_meta_folio_retry(sbi, blk_addr);
entry = f2fs_grab_meta_cache(sbi, blk_addr);
if (IS_ERR(folio))
return;
memcpy(folio_address(folio), src, PAGE_SIZE);
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
memcpy(cache_address(entry), src, F2FS_BLKSIZE(sbi));
f2fs_mark_cache_dirty(entry);
f2fs_put_cache(entry, true);
}
static void write_sum_page(struct f2fs_sb_info *sbi,
static void write_sum_block(struct f2fs_sb_info *sbi,
struct f2fs_summary_block *sum_blk, unsigned int segno)
{
struct folio *folio;
struct f2fs_cached_block *entry;
if (!f2fs_sb_has_packed_ssa(sbi))
return f2fs_update_meta_page(sbi, (void *)sum_blk,
return f2fs_update_meta_block(sbi, (void *)sum_blk,
GET_SUM_BLOCK(sbi, segno));
folio = f2fs_get_sum_folio(sbi, segno);
if (IS_ERR(folio))
entry = f2fs_get_sum_cache(sbi, segno);
if (IS_ERR(entry))
return;
memcpy(SUM_BLK_PAGE_ADDR(sbi, folio, segno), sum_blk,
memcpy(SUM_BLK_ENTRY_ADDR(sbi, entry, segno), sum_blk,
sbi->sum_blocksize);
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
f2fs_mark_cache_dirty(entry);
f2fs_put_cache(entry, true);
}
static void write_current_sum_page(struct f2fs_sb_info *sbi,
static void write_current_sum_block(struct f2fs_sb_info *sbi,
int type, block_t blk_addr)
{
struct curseg_info *curseg = CURSEG_I(sbi, type);
struct folio *folio = f2fs_grab_meta_folio(sbi, blk_addr);
struct f2fs_cached_block *entry = f2fs_grab_meta_cache(sbi, blk_addr);
struct f2fs_summary_block *src = curseg->sum_blk;
struct f2fs_summary_block *dst;
dst = folio_address(folio);
memset(dst, 0, PAGE_SIZE);
dst = cache_address(entry);
memset(dst, 0, sbi->blocksize);
mutex_lock(&curseg->curseg_mutex);
@ -2843,8 +2900,8 @@ static void write_current_sum_page(struct f2fs_sb_info *sbi,
mutex_unlock(&curseg->curseg_mutex);
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
f2fs_mark_cache_dirty(entry);
f2fs_put_cache(entry, true);
}
static int is_next_segment_free(struct f2fs_sb_info *sbi,
@ -2926,7 +2983,7 @@ static int get_new_segment(struct f2fs_sb_info *sbi,
alloc_hint = MAIN_SECS(sbi);
if (pinning) {
max_secno = sbi->pinned_area_max_secno;
max_secno = min(sbi->pinned_area_max_secno, MAIN_SECS(sbi));
hint = 0;
} else if (alloc_policy == ALLOCATE_FORWARD_FROM_HINT &&
hint < alloc_hint) {
@ -3104,7 +3161,7 @@ static int new_curseg(struct f2fs_sb_info *sbi, int type, bool new_sec)
int ret;
if (curseg->inited)
write_sum_page(sbi, curseg->sum_blk, segno);
write_sum_block(sbi, curseg->sum_blk, segno);
segno = __get_next_segno(sbi, type);
ret = get_new_segment(sbi, &segno, new_sec, pinning);
@ -3160,10 +3217,10 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type)
struct curseg_info *curseg = CURSEG_I(sbi, type);
unsigned int new_segno = curseg->next_segno;
struct f2fs_summary_block *sum_node;
struct folio *sum_folio;
struct f2fs_cached_block *entry = NULL;
if (curseg->inited)
write_sum_page(sbi, curseg->sum_blk, curseg->segno);
write_sum_block(sbi, curseg->sum_blk, curseg->segno);
__set_test_and_inuse(sbi, new_segno);
@ -3176,15 +3233,16 @@ static int change_curseg(struct f2fs_sb_info *sbi, int type)
curseg->alloc_type = SSR;
curseg->next_blkoff = __next_free_blkoff(sbi, curseg->segno, 0);
sum_folio = f2fs_get_sum_folio(sbi, new_segno);
if (IS_ERR(sum_folio)) {
/* GC won't be able to use stale summary pages by cp_error */
entry = f2fs_get_sum_cache(sbi, new_segno);
if (IS_ERR(entry)) {
/* GC won't be able to use stale summary blocks by cp_error */
memset(curseg->sum_blk, 0, sbi->sum_entry_size);
return PTR_ERR(sum_folio);
reset_curseg_fields(curseg);
return PTR_ERR(entry);
}
sum_node = SUM_BLK_PAGE_ADDR(sbi, sum_folio, new_segno);
sum_node = SUM_BLK_ENTRY_ADDR(sbi, entry, new_segno);
memcpy(curseg->sum_blk, sum_node, sbi->sum_entry_size);
f2fs_folio_put(sum_folio, true);
f2fs_put_cache(entry, true);
return 0;
}
@ -3271,7 +3329,7 @@ static void __f2fs_save_inmem_curseg(struct f2fs_sb_info *sbi, int type)
goto out;
if (get_valid_blocks(sbi, curseg->segno, false)) {
write_sum_page(sbi, curseg->sum_blk, curseg->segno);
write_sum_block(sbi, curseg->sum_blk, curseg->segno);
} else {
mutex_lock(&DIRTY_I(sbi)->seglist_lock);
__set_test_and_free(sbi, curseg->segno, true);
@ -3592,8 +3650,8 @@ static unsigned int __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range)
{
__u64 start = F2FS_BYTES_TO_BLK(range->start);
__u64 end = start + F2FS_BYTES_TO_BLK(range->len) - 1;
__u64 start = F2FS_BYTES_TO_BLK(sbi, range->start);
__u64 end = start + F2FS_BYTES_TO_BLK(sbi, range->len) - 1;
unsigned int start_segno, end_segno;
block_t start_block, end_block;
struct cp_control cpc;
@ -3624,7 +3682,8 @@ int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range)
}
cpc.reason = CP_DISCARD;
cpc.trim_minlen = max_t(__u64, 1, F2FS_BYTES_TO_BLK(range->minlen));
cpc.trim_minlen = max_t(__u64, 1,
F2FS_BYTES_TO_BLK(sbi, range->minlen));
cpc.trim_start = start_segno;
cpc.trim_end = end_segno;
@ -3658,7 +3717,7 @@ int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range)
start_block, end_block);
out:
if (!err)
range->len = F2FS_BLK_TO_BYTES(trimmed);
range->len = F2FS_BLK_TO_BYTES(sbi, trimmed);
return err;
}
@ -3770,7 +3829,8 @@ static int __get_segment_type_4(struct f2fs_io_info *fio)
else
return CURSEG_COLD_DATA;
} else {
if (IS_DNODE(fio->folio) && is_cold_node(fio->folio))
f2fs_bug_on(fio->sbi, !fio->is_cache);
if (IS_DNODE(fio->sbi, fio->cache_entry) && is_cold_node(fio->sbi, fio->cache_entry))
return CURSEG_WARM_NODE;
else
return CURSEG_COLD_NODE;
@ -3828,9 +3888,9 @@ static int __get_segment_type_6(struct f2fs_io_info *fio)
return f2fs_rw_hint_to_seg_type(F2FS_I_SB(inode),
inode->i_write_hint);
} else {
if (IS_DNODE(fio->folio))
return is_cold_node(fio->folio) ? CURSEG_WARM_NODE :
CURSEG_HOT_NODE;
f2fs_bug_on(fio->sbi, !fio->is_cache);
if (IS_DNODE(fio->sbi, fio->cache_entry))
return is_cold_node(fio->sbi, fio->cache_entry) ? CURSEG_WARM_NODE : CURSEG_HOT_NODE;
return CURSEG_COLD_NODE;
}
}
@ -3897,7 +3957,7 @@ static void f2fs_randomize_chunk(struct f2fs_sb_info *sbi,
get_random_u32_inclusive(1, sbi->max_fragment_hole);
}
int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio,
int f2fs_allocate_data_block(struct f2fs_sb_info *sbi,
block_t old_blkaddr, block_t *new_blkaddr,
struct f2fs_summary *sum, int type,
struct f2fs_io_info *fio)
@ -3966,7 +4026,7 @@ int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio,
if (segment_full) {
if (type == CURSEG_COLD_DATA_PINNED &&
!((curseg->segno + 1) % sbi->segs_per_sec)) {
write_sum_page(sbi, curseg->sum_blk, curseg->segno);
write_sum_block(sbi, curseg->sum_blk, curseg->segno);
reset_curseg_fields(curseg);
goto skip_new_segment;
}
@ -4005,10 +4065,10 @@ int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio,
up_write(&sit_i->sentry_lock);
if (folio && IS_NODESEG(curseg->seg_type)) {
fill_node_footer_blkaddr(folio, NEXT_FREE_BLKADDR(sbi, curseg));
f2fs_inode_chksum_set(sbi, folio);
if (fio && fio->is_cache && IS_NODESEG(curseg->seg_type)) {
fill_node_footer_blkaddr(sbi, fio->cache_entry,
NEXT_FREE_BLKADDR(sbi, curseg));
f2fs_inode_chksum_set(sbi, fio->cache_entry);
}
if (fio) {
@ -4084,7 +4144,7 @@ static int log_type_to_seg_type(enum log_type type)
return seg_type;
}
static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio)
static void do_write_block(struct f2fs_summary *sum, struct f2fs_io_info *fio)
{
struct folio *folio = fio->folio;
enum log_type type = __get_segment_type(fio);
@ -4096,16 +4156,24 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio)
if (keep_order)
f2fs_down_read(&fio->sbi->io_order_lock);
err = f2fs_allocate_data_block(fio->sbi, folio, fio->old_blkaddr,
err = f2fs_allocate_data_block(fio->sbi, fio->old_blkaddr,
&fio->new_blkaddr, sum, type, fio);
if (unlikely(err)) {
f2fs_err_ratelimited(fio->sbi,
"%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d",
__func__, fio->ino, folio->index, type,
fio->old_blkaddr, fio->new_blkaddr, err);
folio_end_writeback(folio);
if (f2fs_in_warm_node_list(folio))
f2fs_del_fsync_node_entry(fio->sbi, folio);
if (fio->is_cache) {
f2fs_err_ratelimited(fio->sbi,
"%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d",
__func__, fio->ino, fio->cache_entry->index, type,
fio->old_blkaddr, fio->new_blkaddr, err);
if (f2fs_in_warm_node_list(fio->sbi, fio->cache_entry))
f2fs_del_fsync_node_entry(fio->sbi, fio->cache_entry);
f2fs_end_cache_writeback(fio->cache_entry);
} else {
f2fs_err_ratelimited(fio->sbi,
"%s Failed to allocate data block, ino:%u, index:%lu, type:%d, old_blkaddr:0x%x, new_blkaddr:0x%x, err:%d",
__func__, fio->ino, folio->index, type,
fio->old_blkaddr, fio->new_blkaddr, err);
folio_end_writeback(folio);
}
f2fs_bug_on(fio->sbi, !is_set_ckpt_flags(fio->sbi,
CP_ERROR_FLAG));
goto out;
@ -4118,7 +4186,10 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio)
f2fs_invalidate_internal_cache(fio->sbi, fio->old_blkaddr, 1);
/* writeout dirty page into bdev */
f2fs_submit_page_write(fio);
if (fio->is_cache)
f2fs_submit_cache_write(fio);
else
f2fs_submit_page_write(fio);
f2fs_update_device_state(fio->sbi, fio->ino, fio->new_blkaddr, 1);
out:
@ -4126,8 +4197,9 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio)
f2fs_up_read(&fio->sbi->io_order_lock);
}
void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio,
enum iostat_type io_type)
void f2fs_do_write_meta_cache(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry,
enum iostat_type io_type)
{
struct f2fs_io_info fio = {
.sbi = sbi,
@ -4135,31 +4207,36 @@ void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct folio *folio,
.temp = HOT,
.op = REQ_OP_WRITE,
.op_flags = REQ_SYNC | REQ_META | REQ_PRIO,
.old_blkaddr = folio->index,
.new_blkaddr = folio->index,
.folio = folio,
.old_blkaddr = entry->index,
.new_blkaddr = entry->index,
.encrypted_page = NULL,
.in_list = 0,
.cache_entry = entry,
.is_cache = 1,
};
if (unlikely(folio->index >= MAIN_BLKADDR(sbi)))
if (unlikely(entry->index >= MAIN_BLKADDR(sbi)))
fio.op_flags &= ~REQ_META;
folio_start_writeback(folio);
f2fs_submit_page_write(&fio);
f2fs_start_cache_writeback(entry);
f2fs_submit_cache_write(&fio);
stat_inc_meta_count(sbi, folio->index);
f2fs_update_iostat(sbi, NULL, io_type, F2FS_BLKSIZE);
stat_inc_meta_count(sbi, entry->index);
f2fs_update_iostat(sbi, NULL, io_type, F2FS_BLKSIZE(sbi));
}
void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio)
void f2fs_do_write_node_cache(unsigned int nid, struct f2fs_io_info *fio)
{
struct f2fs_summary sum;
set_summary(&sum, nid, 0, 0);
do_write_page(&sum, fio);
if (fio->is_cache)
trace_f2fs_write_cache(fio->cache_entry, NODE);
f2fs_update_iostat(fio->sbi, NULL, fio->io_type, F2FS_BLKSIZE);
set_summary(&sum, nid, 0, 0);
do_write_block(&sum, fio);
f2fs_update_iostat(fio->sbi, NULL, fio->io_type,
F2FS_BLKSIZE(fio->sbi));
}
void f2fs_outplace_write_data(struct dnode_of_data *dn,
@ -4172,10 +4249,11 @@ void f2fs_outplace_write_data(struct dnode_of_data *dn,
if (fio->io_type == FS_DATA_IO || fio->io_type == FS_CP_DATA_IO)
f2fs_update_age_extent_cache(dn);
set_summary(&sum, dn->nid, dn->ofs_in_node, fio->version);
do_write_page(&sum, fio);
do_write_block(&sum, fio);
f2fs_update_data_blkaddr(dn, fio->new_blkaddr);
f2fs_update_iostat(sbi, dn->inode, fio->io_type, F2FS_BLKSIZE);
f2fs_update_iostat(sbi, dn->inode, fio->io_type,
F2FS_BLKSIZE(sbi));
}
int f2fs_inplace_write_data(struct f2fs_io_info *fio)
@ -4205,7 +4283,7 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio)
}
if (fio->meta_gc)
f2fs_truncate_meta_inode_pages(sbi, fio->new_blkaddr, 1);
f2fs_truncate_meta_caches(sbi, fio->new_blkaddr, 1);
stat_inc_inplace_blocks(fio->sbi);
@ -4214,10 +4292,11 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio)
else
err = f2fs_submit_page_bio(fio);
if (!err) {
fio->submitted = 1;
f2fs_update_device_state(fio->sbi, fio->ino,
fio->new_blkaddr, 1);
f2fs_update_iostat(fio->sbi, fio_inode(fio),
fio->io_type, F2FS_BLKSIZE);
fio->io_type, F2FS_BLKSIZE(fio->sbi));
}
return err;
@ -4349,14 +4428,13 @@ void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn,
f2fs_update_data_blkaddr(dn, new_addr);
}
void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type,
bool ordered, bool locked)
void f2fs_folio_wait_writeback(struct folio *folio, bool ordered, bool locked)
{
if (folio_test_writeback(folio)) {
struct f2fs_sb_info *sbi = F2FS_F_SB(folio);
/* submit cached LFS IO */
f2fs_submit_merged_write_folio(sbi, folio, type);
f2fs_submit_merged_write_folio(sbi, folio);
/* submit cached IPU IO */
f2fs_submit_merged_ipu_write(sbi, NULL, folio);
if (ordered) {
@ -4371,7 +4449,7 @@ void f2fs_folio_wait_writeback(struct folio *folio, enum page_type type,
void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct folio *cfolio;
struct f2fs_cached_block *entry;
if (!f2fs_meta_inode_gc_required(inode))
return;
@ -4379,11 +4457,12 @@ void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr)
if (!__is_valid_data_blkaddr(blkaddr))
return;
cfolio = filemap_lock_folio(META_MAPPING(sbi), blkaddr);
if (!IS_ERR(cfolio)) {
f2fs_folio_wait_writeback(cfolio, DATA, true, true);
f2fs_folio_put(cfolio, true);
}
entry = f2fs_find_cache(META_CACHE(sbi), blkaddr, 0);
if (IS_ERR(entry))
return;
f2fs_lock_cache(entry);
f2fs_cache_wait_writeback_cond(entry, DATA);
f2fs_put_cache(entry, true);
}
void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr,
@ -4398,7 +4477,7 @@ void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr,
for (i = 0; i < len; i++)
f2fs_wait_on_block_writeback(inode, blkaddr + i);
f2fs_truncate_meta_inode_pages(sbi, blkaddr, len);
f2fs_truncate_meta_caches(sbi, blkaddr, len);
}
static int read_compacted_summaries(struct f2fs_sb_info *sbi)
@ -4406,16 +4485,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi)
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
struct curseg_info *seg_i;
unsigned char *kaddr;
struct folio *folio;
struct f2fs_cached_block *entry;
block_t start;
int i, j, offset;
start = start_sum_block(sbi);
folio = f2fs_get_meta_folio(sbi, start++);
if (IS_ERR(folio))
return PTR_ERR(folio);
kaddr = folio_address(folio);
entry = f2fs_get_meta_cache(sbi, start++);
if (IS_ERR(entry))
return PTR_ERR(entry);
kaddr = cache_address(entry);
/* Step 1: restore nat cache */
seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA);
@ -4452,16 +4531,16 @@ static int read_compacted_summaries(struct f2fs_sb_info *sbi)
SUM_FOOTER_SIZE)
continue;
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
folio = f2fs_get_meta_folio(sbi, start++);
if (IS_ERR(folio))
return PTR_ERR(folio);
kaddr = folio_address(folio);
entry = f2fs_get_meta_cache(sbi, start++);
if (IS_ERR(entry))
return PTR_ERR(entry);
kaddr = cache_address(entry);
offset = 0;
}
}
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
return 0;
}
@ -4470,7 +4549,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
struct f2fs_summary_block *sum;
struct curseg_info *curseg;
struct folio *new;
struct f2fs_cached_block *entry;
unsigned short blk_off;
unsigned int segno = 0;
block_t blk_addr = 0;
@ -4497,10 +4576,10 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
blk_addr = GET_SUM_BLOCK(sbi, segno);
}
new = f2fs_get_meta_folio(sbi, blk_addr);
if (IS_ERR(new))
return PTR_ERR(new);
sum = folio_address(new);
entry = f2fs_get_meta_cache(sbi, blk_addr);
if (IS_ERR(entry))
return PTR_ERR(entry);
sum = cache_address(entry);
if (IS_NODESEG(type)) {
if (__exist_node_summaries(sbi)) {
@ -4537,7 +4616,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
curseg->next_blkoff = blk_off;
mutex_unlock(&curseg->curseg_mutex);
out:
f2fs_folio_put(new, true);
f2fs_put_cache(entry, true);
return err;
}
@ -4549,11 +4628,11 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi)
int err;
if (is_set_ckpt_flags(sbi, CP_COMPACT_SUM_FLAG)) {
int npages = f2fs_npages_for_summary_flush(sbi, true);
int nblocks = f2fs_nblocks_for_summary_flush(sbi, true);
if (npages >= 2)
f2fs_ra_meta_pages(sbi, start_sum_block(sbi), npages,
META_CP, true);
if (nblocks >= 2)
f2fs_ra_meta_caches(sbi, start_sum_block(sbi),
nblocks, META_CP, true);
/* restore for compacted data summary */
err = read_compacted_summaries(sbi);
@ -4563,7 +4642,7 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi)
}
if (__exist_node_summaries(sbi))
f2fs_ra_meta_pages(sbi,
f2fs_ra_meta_caches(sbi,
sum_blk_addr(sbi, NR_CURSEG_PERSIST_TYPE, type),
NR_CURSEG_PERSIST_TYPE - type, META_CP, true);
@ -4586,16 +4665,16 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi)
static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr)
{
struct folio *folio;
struct f2fs_cached_block *entry = NULL;
unsigned char *kaddr;
struct f2fs_summary *summary;
struct curseg_info *seg_i;
int written_size = 0;
int i, j;
folio = f2fs_grab_meta_folio(sbi, blkaddr++);
kaddr = folio_address(folio);
memset(kaddr, 0, PAGE_SIZE);
entry = f2fs_grab_meta_cache(sbi, blkaddr++);
kaddr = cache_address(entry);
memset(kaddr, 0, sbi->blocksize);
/* Step 1: write nat cache */
seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA);
@ -4611,10 +4690,10 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr)
for (i = CURSEG_HOT_DATA; i <= CURSEG_COLD_DATA; i++) {
seg_i = CURSEG_I(sbi, i);
for (j = 0; j < f2fs_curseg_valid_blocks(sbi, i); j++) {
if (!folio) {
folio = f2fs_grab_meta_folio(sbi, blkaddr++);
kaddr = folio_address(folio);
memset(kaddr, 0, PAGE_SIZE);
if (!entry) {
entry = f2fs_grab_meta_cache(sbi, blkaddr++);
kaddr = cache_address(entry);
memset(kaddr, 0, sbi->blocksize);
written_size = 0;
}
summary = (struct f2fs_summary *)(kaddr + written_size);
@ -4625,14 +4704,14 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr)
SUM_FOOTER_SIZE)
continue;
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
folio = NULL;
f2fs_mark_cache_dirty(entry);
f2fs_put_cache(entry, true);
entry = NULL;
}
}
if (folio) {
folio_mark_dirty(folio);
f2fs_folio_put(folio, true);
if (entry) {
f2fs_mark_cache_dirty(entry);
f2fs_put_cache(entry, true);
}
}
@ -4647,7 +4726,7 @@ static void write_normal_summaries(struct f2fs_sb_info *sbi,
end = type + NR_CURSEG_NODE_TYPE;
for (i = type; i < end; i++)
write_current_sum_page(sbi, i, blkaddr + (i - type));
write_current_sum_block(sbi, i, blkaddr + (i - type));
}
void f2fs_write_data_summaries(struct f2fs_sb_info *sbi, block_t start_blk)
@ -4686,29 +4765,29 @@ int f2fs_lookup_journal_in_cursum(struct f2fs_sb_info *sbi,
return -1;
}
static struct folio *get_current_sit_folio(struct f2fs_sb_info *sbi,
static struct f2fs_cached_block *get_current_sit_cache(struct f2fs_sb_info *sbi,
unsigned int segno)
{
return f2fs_get_meta_folio(sbi, current_sit_addr(sbi, segno));
return f2fs_get_meta_cache(sbi, current_sit_addr(sbi, segno));
}
static struct folio *get_next_sit_folio(struct f2fs_sb_info *sbi,
static struct f2fs_cached_block *get_next_sit_cache(struct f2fs_sb_info *sbi,
unsigned int start)
{
struct sit_info *sit_i = SIT_I(sbi);
struct folio *folio;
struct f2fs_cached_block *entry;
pgoff_t src_off, dst_off;
src_off = current_sit_addr(sbi, start);
dst_off = next_sit_addr(sbi, src_off);
folio = f2fs_grab_meta_folio(sbi, dst_off);
seg_info_to_sit_folio(sbi, folio, start);
entry = f2fs_grab_meta_cache(sbi, dst_off);
seg_info_to_sit_block(sbi, entry, start);
folio_mark_dirty(folio);
set_to_next_sit(sit_i, start);
f2fs_mark_cache_dirty(entry);
set_to_next_sit(sbi, sit_i, start);
return folio;
return entry;
}
static struct sit_entry_set *grab_sit_entry_set(void)
@ -4745,10 +4824,11 @@ static void adjust_sit_entry_set(struct sit_entry_set *ses,
list_move_tail(&ses->set_list, head);
}
static void add_sit_entry(unsigned int segno, struct list_head *head)
static void add_sit_entry(struct f2fs_sb_info *sbi, unsigned int segno,
struct list_head *head)
{
struct sit_entry_set *ses;
unsigned int start_segno = START_SEGNO(segno);
unsigned int start_segno = f2fs_start_segno(sbi, segno);
list_for_each_entry(ses, head, set_list) {
if (ses->start_segno == start_segno) {
@ -4773,7 +4853,7 @@ static void add_sits_in_set(struct f2fs_sb_info *sbi)
unsigned int segno;
for_each_set_bit(segno, bitmap, MAIN_SEGS(sbi))
add_sit_entry(segno, set_list);
add_sit_entry(sbi, segno, set_list);
}
static void remove_sits_in_journal(struct f2fs_sb_info *sbi)
@ -4791,7 +4871,7 @@ static void remove_sits_in_journal(struct f2fs_sb_info *sbi)
dirtied = __mark_sit_entry_dirty(sbi, segno);
if (!dirtied)
add_sit_entry(segno, &SM_I(sbi)->sit_entry_set);
add_sit_entry(sbi, segno, &SM_I(sbi)->sit_entry_set);
}
update_sits_in_cursum(journal, -i);
up_write(&curseg->journal_rwsem);
@ -4814,7 +4894,7 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
down_write(&sit_i->sentry_lock);
if (!sit_i->dirty_sentries)
if (!sit_i->dirty_sentries && to_journal)
goto out;
/*
@ -4838,10 +4918,11 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
* #2, flush sit entries to sit page.
*/
list_for_each_entry_safe(ses, tmp, head, set_list) {
struct folio *folio = NULL;
struct f2fs_cached_block *entry = NULL;
struct f2fs_sit_block *raw_sit = NULL;
unsigned int start_segno = ses->start_segno;
unsigned int end = min(start_segno + SIT_ENTRY_PER_BLOCK,
unsigned int end = min(start_segno + SIT_ENTRY_PER_BLOCK(sbi),
(unsigned long)MAIN_SEGS(sbi));
unsigned int segno = start_segno;
@ -4853,8 +4934,8 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
if (to_journal) {
down_write(&curseg->journal_rwsem);
} else {
folio = get_next_sit_folio(sbi, start_segno);
raw_sit = folio_address(folio);
entry = get_next_sit_cache(sbi, start_segno);
raw_sit = cache_address(entry);
}
/* flush dirty sit entries in region of current sit set */
@ -4899,7 +4980,7 @@ void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
if (to_journal)
up_write(&curseg->journal_rwsem);
else
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
f2fs_bug_on(sbi, ses->entry_cnt);
release_sit_entry_set(ses);
@ -5006,7 +5087,7 @@ static int build_sit_info(struct f2fs_sb_info *sbi)
sit_i->written_valid_blocks = 0;
sit_i->bitmap_size = sit_bitmap_size;
sit_i->dirty_sentries = 0;
sit_i->sents_per_block = SIT_ENTRY_PER_BLOCK;
sit_i->sents_per_block = SIT_ENTRY_PER_BLOCK(sbi);
sit_i->elapsed_time = le64_to_cpu(sbi->ckpt->elapsed_time);
sit_i->mounted_time = ktime_get_boottime_seconds();
init_rwsem(&sit_i->sentry_lock);
@ -5090,7 +5171,7 @@ static int build_sit_entries(struct f2fs_sb_info *sbi)
block_t sit_valid_blocks[2] = {0, 0};
do {
readed = f2fs_ra_meta_pages(sbi, start_blk, BIO_MAX_VECS,
readed = f2fs_ra_meta_caches(sbi, start_blk, BIO_MAX_VECS,
META_SIT, true);
start = start_blk * sit_i->sents_per_block;
@ -5098,15 +5179,15 @@ static int build_sit_entries(struct f2fs_sb_info *sbi)
for (; start < end && start < MAIN_SEGS(sbi); start++) {
struct f2fs_sit_block *sit_blk;
struct folio *folio;
struct f2fs_cached_block *entry;
se = &sit_i->sentries[start];
folio = get_current_sit_folio(sbi, start);
if (IS_ERR(folio))
return PTR_ERR(folio);
sit_blk = folio_address(folio);
entry = get_current_sit_cache(sbi, start);
if (IS_ERR(entry))
return PTR_ERR(entry);
sit_blk = cache_address(entry);
sit = sit_blk->entries[SIT_ENTRY_OFFSET(sit_i, start)];
f2fs_folio_put(folio, true);
f2fs_put_cache(entry, true);
err = check_block_count(sbi, start, &sit);
if (err)

98
fs/f2fs/segment.h
View File

@ -91,30 +91,46 @@ static inline void sanity_check_seg_type(struct f2fs_sb_info *sbi,
#define GET_ZONE_FROM_SEG(sbi, segno) \
GET_ZONE_FROM_SEC(sbi, GET_SEC_FROM_SEG(sbi, segno))
static inline void f2fs_adjust_pinned_area_boundary(struct f2fs_sb_info *sbi)
{
sbi->pinned_area_max_secno = MAIN_SECS(sbi);
if (f2fs_sb_has_blkzoned(sbi) && sbi->first_seq_zone_segno != NULL_SEGNO)
sbi->pinned_area_max_secno = min(sbi->pinned_area_max_secno,
GET_SEC_FROM_SEG(sbi, sbi->first_seq_zone_segno));
if (F2FS_OPTION(sbi).resizable_tail_secno)
sbi->pinned_area_max_secno = min(sbi->pinned_area_max_secno,
MAIN_SECS(sbi) - F2FS_OPTION(sbi).resizable_tail_secno);
}
#define GET_SUM_BLOCK(sbi, segno) \
(SM_I(sbi)->ssa_blkaddr + (segno / (sbi)->sums_per_block))
#define GET_SUM_BLKOFF(sbi, segno) (segno % (sbi)->sums_per_block)
#define SUM_BLK_PAGE_ADDR(sbi, folio, segno) \
(folio_address(folio) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize)
#define SUM_BLK_ENTRY_ADDR(sbi, entry, segno) \
(cache_address(entry) + GET_SUM_BLKOFF(sbi, segno) * (sbi)->sum_blocksize)
#define GET_SUM_TYPE(footer) ((footer)->entry_type)
#define SET_SUM_TYPE(footer, type) ((footer)->entry_type = (type))
#define SIT_ENTRY_OFFSET(sit_i, segno) \
((segno) % (sit_i)->sents_per_block)
#define SIT_BLOCK_OFFSET(segno) \
((segno) / SIT_ENTRY_PER_BLOCK)
#define START_SEGNO(segno) \
(SIT_BLOCK_OFFSET(segno) * SIT_ENTRY_PER_BLOCK)
#define SIT_BLOCK_OFFSET(sbi, segno) \
((segno) / SIT_ENTRY_PER_BLOCK(sbi))
static inline unsigned int
f2fs_start_segno(struct f2fs_sb_info *sbi, unsigned int segno)
{
return SIT_BLOCK_OFFSET(sbi, segno) * SIT_ENTRY_PER_BLOCK(sbi);
}
#define SIT_BLK_CNT(sbi) \
DIV_ROUND_UP(MAIN_SEGS(sbi), SIT_ENTRY_PER_BLOCK)
DIV_ROUND_UP(MAIN_SEGS(sbi), SIT_ENTRY_PER_BLOCK(sbi))
#define f2fs_bitmap_size(nr) \
(BITS_TO_LONGS(nr) * sizeof(unsigned long))
#define SECTOR_FROM_BLOCK(blk_addr) \
(((sector_t)blk_addr) << F2FS_LOG_SECTORS_PER_BLOCK)
#define SECTOR_TO_BLOCK(sectors) \
((sectors) >> F2FS_LOG_SECTORS_PER_BLOCK)
#define SECTOR_FROM_BLOCK(sbi, blk_addr) \
(((sector_t)blk_addr) << F2FS_LOG_SECTORS_PER_BLOCK(sbi))
#define SECTOR_TO_BLOCK(sbi, sectors) \
((sectors) >> F2FS_LOG_SECTORS_PER_BLOCK(sbi))
/*
* In the victim_sel_policy->alloc_mode, there are three block allocation modes.
@ -418,18 +434,18 @@ static inline void __seg_info_to_raw_sit(struct seg_entry *se,
rs->mtime = cpu_to_le64(se->mtime);
}
static inline void seg_info_to_sit_folio(struct f2fs_sb_info *sbi,
struct folio *folio, unsigned int start)
static inline void seg_info_to_sit_block(struct f2fs_sb_info *sbi,
struct f2fs_cached_block *entry, unsigned int start)
{
struct f2fs_sit_block *raw_sit;
struct seg_entry *se;
struct f2fs_sit_entry *rs;
unsigned int end = min(start + SIT_ENTRY_PER_BLOCK,
unsigned int end = min(start + SIT_ENTRY_PER_BLOCK(sbi),
(unsigned long)MAIN_SEGS(sbi));
int i;
raw_sit = folio_address(folio);
memset(raw_sit, 0, PAGE_SIZE);
raw_sit = cache_address(entry);
memset(raw_sit, 0, F2FS_BLKSIZE(sbi));
for (i = 0; i < end - start; i++) {
rs = &raw_sit->entries[i];
se = get_seg_entry(sbi, start + i);
@ -651,15 +667,15 @@ static inline void get_additional_blocks_required(struct f2fs_sb_info *sbi,
*/
static inline int __get_secs_required(struct f2fs_sb_info *sbi)
{
unsigned int total_node_blocks = get_pages(sbi, F2FS_DIRTY_NODES) +
get_pages(sbi, F2FS_DIRTY_DENTS) +
get_pages(sbi, F2FS_DIRTY_IMETA);
unsigned int total_dent_blocks = get_pages(sbi, F2FS_DIRTY_DENTS);
unsigned int total_node_blocks = get_nr_caches(sbi, F2FS_DIRTY_NODES) +
get_nr_caches(sbi, F2FS_DIRTY_DENTS) +
get_nr_caches(sbi, F2FS_DIRTY_IMETA);
unsigned int total_dent_blocks = get_nr_caches(sbi, F2FS_DIRTY_DENTS);
unsigned int total_data_blocks = 0;
bool separate_dent = true;
if (f2fs_lfs_mode(sbi))
total_data_blocks = get_pages(sbi, F2FS_DIRTY_DATA);
total_data_blocks = get_nr_caches(sbi, F2FS_DIRTY_DATA);
/*
* When active_logs != 4, dentry blocks and data blocks can be
@ -869,7 +885,7 @@ static inline pgoff_t current_sit_addr(struct f2fs_sb_info *sbi,
unsigned int start)
{
struct sit_info *sit_i = SIT_I(sbi);
unsigned int offset = SIT_BLOCK_OFFSET(start);
unsigned int offset = SIT_BLOCK_OFFSET(sbi, start);
block_t blk_addr = sit_i->sit_base_addr + offset;
f2fs_bug_on(sbi, !valid_main_segno(sbi, start));
@ -894,9 +910,10 @@ static inline pgoff_t next_sit_addr(struct f2fs_sb_info *sbi,
return block_addr + sit_i->sit_base_addr;
}
static inline void set_to_next_sit(struct sit_info *sit_i, unsigned int start)
static inline void set_to_next_sit(struct f2fs_sb_info *sbi,
struct sit_info *sit_i, unsigned int start)
{
unsigned int block_off = SIT_BLOCK_OFFSET(start);
unsigned int block_off = SIT_BLOCK_OFFSET(sbi, start);
f2fs_change_bit(block_off, sit_i->sit_bitmap);
}
@ -971,13 +988,13 @@ static inline bool sec_usage_check(struct f2fs_sb_info *sbi, unsigned int secno)
}
/*
* It is very important to gather dirty pages and write at once, so that we can
* It is very important to gather dirty blocks and write at once, so that we can
* submit a big bio without interfering other data writes.
* By default, 512 pages for directory data,
* 512 pages (2MB) * 8 for nodes, and
* 256 pages * 8 for meta are set.
* By default, 512 blocks for directory data,
* 512 blocks (2MB) * 8 for nodes, and
* 256 blocks * 8 for meta are set.
*/
static inline int nr_pages_to_skip(struct f2fs_sb_info *sbi, int type)
static inline int nr_caches_to_skip(struct f2fs_sb_info *sbi, int type)
{
if (bdi_wb_dirty_exceeded(sbi->sb->s_bdi))
return 0;
@ -993,23 +1010,24 @@ static inline int nr_pages_to_skip(struct f2fs_sb_info *sbi, int type)
}
/*
* When writing pages, it'd better align nr_to_write for segment size.
* When writing cache asynchronously, align nr_to_write to BIO_MAX_VECS.
*/
static inline long nr_pages_to_write(struct f2fs_sb_info *sbi, int type,
struct writeback_control *wbc)
static inline long adjust_flush_cache_number(struct f2fs_sb_info *sbi, int type)
{
long nr_to_write, desired;
long nr_to_write;
if (wbc->sync_mode != WB_SYNC_NONE)
switch (type) {
case META:
nr_to_write = BIO_MAX_VECS;
break;
case NODE:
nr_to_write = BIO_MAX_VECS << 1;
break;
default:
f2fs_bug_on(sbi, 1);
return 0;
nr_to_write = wbc->nr_to_write;
desired = BIO_MAX_VECS;
if (type == NODE)
desired <<= 1;
wbc->nr_to_write = desired;
return desired - nr_to_write;
}
return nr_to_write;
}
static inline void wake_up_discard_thread(struct f2fs_sb_info *sbi, bool force)

16
fs/f2fs/shrinker.c
View File

@ -23,7 +23,7 @@ static unsigned long __count_nat_entries(struct f2fs_sb_info *sbi)
static unsigned long __count_free_nids(struct f2fs_sb_info *sbi)
{
long count = NM_I(sbi)->nid_cnt[FREE_NID] - MAX_FREE_NIDS;
long count = NM_I(sbi)->nid_cnt[FREE_NID] - MAX_FREE_NIDS(sbi);
return count > 0 ? count : 0;
}
@ -37,6 +37,13 @@ static unsigned long __count_extent_cache(struct f2fs_sb_info *sbi,
atomic_read(&eti->total_ext_node);
}
static unsigned long __count_cache(struct f2fs_sb_info *sbi)
{
return META_CACHE(sbi)->num_entries +
NODE_CACHE(sbi)->num_entries +
COMPRESS_CACHE(sbi)->num_entries;
}
unsigned long f2fs_shrink_count(struct shrinker *shrink,
struct shrink_control *sc)
{
@ -68,6 +75,9 @@ unsigned long f2fs_shrink_count(struct shrinker *shrink,
/* count free nids cache entries */
count += __count_free_nids(sbi);
/* count generic cache entries */
count += __count_cache(sbi);
spin_lock(&f2fs_list_lock);
p = p->next;
mutex_unlock(&sbi->umount_mutex);
@ -120,6 +130,10 @@ unsigned long f2fs_shrink_scan(struct shrinker *shrink,
if (freed < nr)
freed += f2fs_try_to_free_nids(sbi, nr - freed);
/* shrink generic cache entries */
if (freed < nr)
freed += f2fs_shrink_cache(sbi, nr - freed);
spin_lock(&f2fs_list_lock);
p = p->next;
list_move_tail(&sbi->s_list, &f2fs_list);

451
fs/f2fs/super.c
View File

File diff suppressed because it is too large Load Diff

44
fs/f2fs/sysfs.c
View File

@ -40,6 +40,7 @@ enum {
RESERVED_BLOCKS, /* struct f2fs_sb_info */
CPRC_INFO, /* struct ckpt_req_control */
ATGC_INFO, /* struct atgc_management */
WB_THREAD, /* struct f2fs_cache_kthread */
};
static const char *gc_mode_names[MAX_GC_MODE] = {
@ -98,6 +99,8 @@ static unsigned char *__struct_ptr(struct f2fs_sb_info *sbi, int struct_type)
return (unsigned char *)&sbi->cprc_info;
else if (struct_type == ATGC_INFO)
return (unsigned char *)&sbi->am;
else if (struct_type == WB_THREAD)
return (unsigned char *)&sbi->cache_thread;
return NULL;
}
@ -995,6 +998,13 @@ static ssize_t __sbi_store(struct f2fs_attr *a,
return count;
}
if (!strcmp(a->attr.name, "cache_wb_interval")) {
if (t < MIN_DIRTY_CACHE_TIMEOUT || t > MAX_DIRTY_CACHE_TIMEOUT)
return -EINVAL;
sbi->cache_thread.cache_wb_interval = t;
return count;
}
__sbi_store_value(a, sbi, ptr + a->offset, t);
return count;
@ -1008,7 +1018,10 @@ static ssize_t f2fs_sbi_store(struct f2fs_attr *a,
bool gc_entry = (!strcmp(a->attr.name, "gc_urgent") ||
a->struct_type == GC_THREAD);
bool thread_entry = !strcmp(a->attr.name, "ckpt_thread_ioprio") ||
!strcmp(a->attr.name, "critical_task_priority");
!strcmp(a->attr.name, "critical_task_priority") ||
!strcmp(a->attr.name, "cache_wb_interval") ||
!strcmp(a->attr.name, "cache_wb_dirty_threshold") ||
!strcmp(a->attr.name, "cache_wb_total_threshold");
if (gc_entry || thread_entry) {
if (!down_read_trylock(&sbi->sb->s_umount))
@ -1218,6 +1231,9 @@ static struct f2fs_attr f2fs_attr_##name = __ATTR(name, 0444, name##_show, NULL)
#define ATGC_INFO_RW_ATTR(name, elname) \
F2FS_RW_ATTR(ATGC_INFO, atgc_management, name, elname)
#define WB_THREAD_RW_ATTR(name, elname) \
F2FS_RW_ATTR(WB_THREAD, f2fs_cache_kthread, name, elname)
/* GC_THREAD ATTR */
GC_THREAD_RW_ATTR(gc_urgent_sleep_time, urgent_sleep_time);
GC_THREAD_RW_ATTR(gc_min_sleep_time, min_sleep_time);
@ -1254,7 +1270,7 @@ DCC_INFO_GENERAL_RW_ATTR(discard_io_aware);
/* NM_INFO ATTR */
NM_INFO_RW_ATTR(max_roll_forward_node_blocks, max_rf_node_blocks);
NM_INFO_GENERAL_RW_ATTR(ram_thresh);
NM_INFO_GENERAL_RW_ATTR(ra_nid_pages);
NM_INFO_RW_ATTR(ra_nid_pages, ra_nid_blocks);
NM_INFO_GENERAL_RW_ATTR(dirty_nats_ratio);
/* F2FS_SBI ATTR */
@ -1269,6 +1285,7 @@ F2FS_SBI_RW_ATTR(umount_discard_timeout, interval_time[UMOUNT_DISCARD_TIMEOUT]);
F2FS_SBI_RW_ATTR(gc_pin_file_thresh, gc_pin_file_threshold);
F2FS_SBI_RW_ATTR(gc_reclaimed_segments, gc_reclaimed_segs);
F2FS_SBI_RW_ATTR(max_atc_write_bio_size, max_atc_write_bio_size);
F2FS_SBI_RW_ATTR(max_atc_write_bio_entry_cnt, max_atc_write_bio_entry_cnt);
F2FS_SBI_GENERAL_RW_ATTR(max_victim_search);
F2FS_SBI_GENERAL_RW_ATTR(migration_granularity);
F2FS_SBI_GENERAL_RW_ATTR(migration_window_granularity);
@ -1347,6 +1364,11 @@ ATGC_INFO_RW_ATTR(atgc_candidate_count, max_candidate_count);
ATGC_INFO_RW_ATTR(atgc_age_weight, age_weight);
ATGC_INFO_RW_ATTR(atgc_age_threshold, age_threshold);
/* WB_THREAD ATTR */
WB_THREAD_RW_ATTR(cache_wb_interval, cache_wb_interval);
WB_THREAD_RW_ATTR(cache_wb_dirty_threshold, cache_wb_dirty_threshold);
WB_THREAD_RW_ATTR(cache_wb_total_threshold, cache_wb_total_threshold);
F2FS_GENERAL_RO_ATTR(dirty_segments);
F2FS_GENERAL_RO_ATTR(free_segments);
F2FS_GENERAL_RO_ATTR(ovp_segments);
@ -1404,6 +1426,8 @@ F2FS_FEATURE_RO_ATTR(linear_lookup);
#endif
F2FS_FEATURE_RO_ATTR(packed_ssa);
F2FS_FEATURE_RO_ATTR(fserror);
F2FS_FEATURE_RO_ATTR(metadata_cache);
F2FS_FEATURE_RO_ATTR(resizable_tail);
#define ATTR_LIST(name) (&f2fs_attr_##name.attr)
static struct attribute *f2fs_attrs[] = {
@ -1514,6 +1538,7 @@ static struct attribute *f2fs_attrs[] = {
ATTR_LIST(gc_segment_mode),
ATTR_LIST(gc_reclaimed_segments),
ATTR_LIST(max_atc_write_bio_size),
ATTR_LIST(max_atc_write_bio_entry_cnt),
ATTR_LIST(max_fragment_chunk),
ATTR_LIST(max_fragment_hole),
ATTR_LIST(current_atomic_write),
@ -1533,6 +1558,9 @@ static struct attribute *f2fs_attrs[] = {
ATTR_LIST(lock_duration_priority),
ATTR_LIST(adjust_lock_priority),
ATTR_LIST(critical_task_priority),
ATTR_LIST(cache_wb_interval),
ATTR_LIST(cache_wb_dirty_threshold),
ATTR_LIST(cache_wb_total_threshold),
NULL,
};
ATTRIBUTE_GROUPS(f2fs);
@ -1574,6 +1602,8 @@ static struct attribute *f2fs_feat_attrs[] = {
#endif
BASE_ATTR_LIST(packed_ssa),
BASE_ATTR_LIST(fserror),
BASE_ATTR_LIST(metadata_cache),
BASE_ATTR_LIST(resizable_tail),
NULL,
};
ATTRIBUTE_GROUPS(f2fs_feat);
@ -1871,8 +1901,8 @@ static int __maybe_unused disk_map_seq_show(struct seq_file *seq,
struct f2fs_sb_info *sbi = F2FS_SB(sb);
int i;
seq_printf(seq, "Address Layout : %5luB Block address (# of Segments)\n",
F2FS_BLKSIZE);
seq_printf(seq, "Address Layout : %5uB Block address (# of Segments)\n",
F2FS_BLKSIZE(sbi));
seq_printf(seq, " SB : %12s\n", "0/1024B");
seq_printf(seq, " seg0_blkaddr : 0x%010x\n", SEG0_BLKADDR(sbi));
seq_printf(seq, " Checkpoint : 0x%010x (%10d)\n",
@ -1889,13 +1919,13 @@ static int __maybe_unused disk_map_seq_show(struct seq_file *seq,
seq_printf(seq, " Main : 0x%010x (%10d)\n",
SM_I(sbi)->main_blkaddr,
le32_to_cpu(F2FS_RAW_SUPER(sbi)->segment_count_main));
seq_printf(seq, " Block size : %12lu KB\n", F2FS_BLKSIZE >> 10);
seq_printf(seq, " Block size : %12u KB\n", F2FS_BLKSIZE(sbi) >> 10);
seq_printf(seq, " Segment size : %12d MB\n",
(BLKS_PER_SEG(sbi) << (F2FS_BLKSIZE_BITS - 10)) >> 10);
(BLKS_PER_SEG(sbi) << (F2FS_BLKSIZE_BITS(sbi) - 10)) >> 10);
seq_printf(seq, " Segs/Sections : %12d\n",
SEGS_PER_SEC(sbi));
seq_printf(seq, " Section size : %12d MB\n",
(BLKS_PER_SEC(sbi) << (F2FS_BLKSIZE_BITS - 10)) >> 10);
(BLKS_PER_SEC(sbi) << (F2FS_BLKSIZE_BITS(sbi) - 10)) >> 10);
seq_printf(seq, " # of Sections : %12d\n",
le32_to_cpu(F2FS_RAW_SUPER(sbi)->section_count));

6
fs/f2fs/verity.c
View File

@ -47,7 +47,8 @@ static int pagecache_write(struct inode *inode, const void *buf, size_t count,
struct address_space *mapping = inode->i_mapping;
const struct address_space_operations *aops = mapping->a_ops;
if (pos + count > F2FS_BLK_TO_BYTES(max_file_blocks(inode)))
if (pos + count > F2FS_BLK_TO_BYTES(F2FS_I_SB(inode),
max_file_blocks(F2FS_I_SB(inode), inode)))
return -EFBIG;
while (count) {
@ -211,7 +212,8 @@ static int f2fs_get_verity_descriptor(struct inode *inode, void *buf,
/* Get the descriptor */
if (pos + size < pos ||
pos + size > F2FS_BLK_TO_BYTES(max_file_blocks(inode)) ||
pos + size > F2FS_BLK_TO_BYTES(F2FS_I_SB(inode),
max_file_blocks(F2FS_I_SB(inode), inode)) ||
pos < f2fs_verity_metadata_pos(inode) || size > INT_MAX) {
f2fs_warn(F2FS_I_SB(inode), "invalid verity xattr");
f2fs_handle_error(F2FS_I_SB(inode),

133
fs/f2fs/xattr.c
View File

@ -138,7 +138,7 @@ static int f2fs_xattr_advise_set(const struct xattr_handler *handler,
#ifdef CONFIG_F2FS_FS_SECURITY
static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array,
void *folio)
void *fs_data)
{
const struct xattr *xattr;
int err = 0;
@ -146,7 +146,7 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array,
for (xattr = xattr_array; xattr->name != NULL; xattr++) {
err = f2fs_setxattr(inode, F2FS_XATTR_INDEX_SECURITY,
xattr->name, xattr->value,
xattr->value_len, folio, 0);
xattr->value_len, fs_data, 0);
if (err < 0)
break;
}
@ -154,10 +154,10 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array,
}
int f2fs_init_security(struct inode *inode, struct inode *dir,
const struct qstr *qstr, struct folio *ifolio)
const struct qstr *qstr, struct f2fs_cached_block *ientry)
{
return security_inode_init_security(inode, dir, qstr,
f2fs_initxattrs, ifolio);
f2fs_initxattrs, ientry);
}
#endif
@ -273,25 +273,25 @@ static struct f2fs_xattr_entry *__find_inline_xattr(struct inode *inode,
return entry;
}
static int read_inline_xattr(struct inode *inode, struct folio *ifolio,
static int read_inline_xattr(struct inode *inode, struct f2fs_cached_block *ientry,
void *txattr_addr)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
unsigned int inline_size = inline_xattr_size(inode);
struct folio *folio = NULL;
struct f2fs_cached_block *in_entry = NULL;
void *inline_addr;
if (ifolio) {
inline_addr = inline_xattr_addr(inode, ifolio);
if (ientry) {
inline_addr = inline_xattr_addr(inode, ientry);
} else {
folio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(folio))
return PTR_ERR(folio);
in_entry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(in_entry))
return PTR_ERR(in_entry);
inline_addr = inline_xattr_addr(inode, folio);
inline_addr = inline_xattr_addr(inode, in_entry);
}
memcpy(txattr_addr, inline_addr, inline_size);
f2fs_folio_put(folio, true);
f2fs_put_cache(in_entry, true);
return 0;
}
@ -301,22 +301,23 @@ static int read_xattr_block(struct inode *inode, void *txattr_addr)
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
nid_t xnid = F2FS_I(inode)->i_xattr_nid;
unsigned int inline_size = inline_xattr_size(inode);
struct folio *xfolio;
struct f2fs_cached_block *xentry;
void *xattr_addr;
/* The inode already has an extended attribute block. */
xfolio = f2fs_get_xnode_folio(sbi, xnid);
if (IS_ERR(xfolio))
return PTR_ERR(xfolio);
xentry = f2fs_get_xnode_cache(sbi, xnid);
if (IS_ERR(xentry))
return PTR_ERR(xentry);
xattr_addr = folio_address(xfolio);
memcpy(txattr_addr + inline_size, xattr_addr, VALID_XATTR_BLOCK_SIZE);
f2fs_folio_put(xfolio, true);
xattr_addr = cache_address(xentry);
memcpy(txattr_addr + inline_size, xattr_addr,
VALID_XATTR_BLOCK_SIZE(inode));
f2fs_put_cache(xentry, true);
return 0;
}
static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio,
static int lookup_all_xattrs(struct inode *inode, struct f2fs_cached_block *ientry,
unsigned int index, unsigned int len,
const char *name, struct f2fs_xattr_entry **xe,
void **base_addr, int *base_size,
@ -340,7 +341,7 @@ static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio,
/* read from inline xattr */
if (inline_size) {
err = read_inline_xattr(inode, ifolio, txattr_addr);
err = read_inline_xattr(inode, ientry, txattr_addr);
if (err)
goto out;
@ -388,12 +389,12 @@ static int lookup_all_xattrs(struct inode *inode, struct folio *ifolio,
return err;
}
static int read_all_xattrs(struct inode *inode, struct folio *ifolio,
static int read_all_xattrs(struct inode *inode, struct f2fs_cached_block *ientry,
void **base_addr)
{
struct f2fs_xattr_header *header;
nid_t xnid = F2FS_I(inode)->i_xattr_nid;
unsigned int size = VALID_XATTR_BLOCK_SIZE;
unsigned int size = VALID_XATTR_BLOCK_SIZE(inode);
unsigned int inline_size = inline_xattr_size(inode);
void *txattr_addr;
int err;
@ -405,7 +406,7 @@ static int read_all_xattrs(struct inode *inode, struct folio *ifolio,
/* read from inline xattr */
if (inline_size) {
err = read_inline_xattr(inode, ifolio, txattr_addr);
err = read_inline_xattr(inode, ientry, txattr_addr);
if (err)
goto fail;
}
@ -432,14 +433,14 @@ static int read_all_xattrs(struct inode *inode, struct folio *ifolio,
}
static inline int write_all_xattrs(struct inode *inode, __u32 hsize,
void *txattr_addr, struct folio *ifolio)
void *txattr_addr, struct f2fs_cached_block *ientry)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
size_t inline_size = inline_xattr_size(inode);
struct folio *in_folio = NULL;
struct f2fs_cached_block *in_entry = NULL;
void *xattr_addr;
void *inline_addr = NULL;
struct folio *xfolio;
struct f2fs_cached_block *xentry;
nid_t new_nid = 0;
int err = 0;
@ -449,75 +450,75 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize,
/* write to inline xattr */
if (inline_size) {
if (ifolio) {
inline_addr = inline_xattr_addr(inode, ifolio);
if (ientry) {
inline_addr = inline_xattr_addr(inode, ientry);
} else {
in_folio = f2fs_get_inode_folio(sbi, inode->i_ino);
if (IS_ERR(in_folio)) {
in_entry = f2fs_get_inode_cache(sbi, inode->i_ino);
if (IS_ERR(in_entry)) {
f2fs_alloc_nid_failed(sbi, new_nid);
return PTR_ERR(in_folio);
return PTR_ERR(in_entry);
}
inline_addr = inline_xattr_addr(inode, in_folio);
inline_addr = inline_xattr_addr(inode, in_entry);
}
f2fs_folio_wait_writeback(ifolio ? ifolio : in_folio,
NODE, true, true);
f2fs_cache_wait_writeback(ientry ? ientry : in_entry);
/* no need to use xattr node block */
if (hsize <= inline_size) {
err = f2fs_truncate_xattr_node(inode);
f2fs_alloc_nid_failed(sbi, new_nid);
if (err) {
f2fs_folio_put(in_folio, true);
f2fs_put_cache(in_entry, true);
return err;
}
memcpy(inline_addr, txattr_addr, inline_size);
folio_mark_dirty(ifolio ? ifolio : in_folio);
f2fs_mark_cache_dirty(ientry ? ientry : in_entry);
goto in_page_out;
}
}
/* write to xattr node block */
if (F2FS_I(inode)->i_xattr_nid) {
xfolio = f2fs_get_xnode_folio(sbi, F2FS_I(inode)->i_xattr_nid);
if (IS_ERR(xfolio)) {
err = PTR_ERR(xfolio);
xentry = f2fs_get_xnode_cache(sbi, F2FS_I(inode)->i_xattr_nid);
if (IS_ERR(xentry)) {
err = PTR_ERR(xentry);
f2fs_alloc_nid_failed(sbi, new_nid);
goto in_page_out;
}
f2fs_bug_on(sbi, new_nid);
f2fs_folio_wait_writeback(xfolio, NODE, true, true);
f2fs_cache_wait_writeback(xentry);
} else {
struct dnode_of_data dn;
set_new_dnode(&dn, inode, NULL, NULL, new_nid);
xfolio = f2fs_new_node_folio(&dn, XATTR_NODE_OFFSET);
if (IS_ERR(xfolio)) {
err = PTR_ERR(xfolio);
xentry = f2fs_new_node_cache(&dn, XATTR_NODE_OFFSET);
if (IS_ERR(xentry)) {
err = PTR_ERR(xentry);
f2fs_alloc_nid_failed(sbi, new_nid);
goto in_page_out;
}
f2fs_alloc_nid_done(sbi, new_nid);
}
xattr_addr = folio_address(xfolio);
xattr_addr = cache_address(xentry);
if (inline_size)
memcpy(inline_addr, txattr_addr, inline_size);
memcpy(xattr_addr, txattr_addr + inline_size, VALID_XATTR_BLOCK_SIZE);
memcpy(xattr_addr, txattr_addr + inline_size,
VALID_XATTR_BLOCK_SIZE(inode));
if (inline_size)
folio_mark_dirty(ifolio ? ifolio : in_folio);
folio_mark_dirty(xfolio);
f2fs_mark_cache_dirty(ientry ? ientry : in_entry);
f2fs_mark_cache_dirty(xentry);
f2fs_folio_put(xfolio, true);
f2fs_put_cache(xentry, true);
in_page_out:
f2fs_folio_put(in_folio, true);
f2fs_put_cache(in_entry, true);
return err;
}
int f2fs_getxattr(struct inode *inode, int index, const char *name,
void *buffer, size_t buffer_size, struct folio *ifolio)
void *buffer, size_t buffer_size, struct f2fs_cached_block *ientry)
{
struct f2fs_xattr_entry *entry = NULL;
struct f2fs_xattr_entry *xe = NULL;
int error;
unsigned int size, len;
void *base_addr = NULL;
@ -531,16 +532,16 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name,
if (len > F2FS_NAME_LEN)
return -ERANGE;
if (!ifolio)
if (!ientry)
f2fs_down_read(&F2FS_I(inode)->i_xattr_sem);
error = lookup_all_xattrs(inode, ifolio, index, len, name,
&entry, &base_addr, &base_size, &is_inline);
if (!ifolio)
error = lookup_all_xattrs(inode, ientry, index, len, name,
&xe, &base_addr, &base_size, &is_inline);
if (!ientry)
f2fs_up_read(&F2FS_I(inode)->i_xattr_sem);
if (error)
return error;
size = le16_to_cpu(entry->e_value_size);
size = le16_to_cpu(xe->e_value_size);
if (buffer && size > buffer_size) {
error = -ERANGE;
@ -548,7 +549,7 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name,
}
if (buffer) {
char *pval = entry->e_name + entry->e_name_len;
char *pval = xe->e_name + xe->e_name_len;
if (base_size - (pval - (char *)base_addr) < size) {
error = -ERANGE;
@ -632,7 +633,7 @@ static bool f2fs_xattr_value_same(struct f2fs_xattr_entry *entry,
static int __f2fs_setxattr(struct inode *inode, int index,
const char *name, const void *value, size_t size,
struct folio *ifolio, int flags)
struct f2fs_cached_block *ientry, int flags)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_xattr_entry *here, *last;
@ -656,7 +657,7 @@ static int __f2fs_setxattr(struct inode *inode, int index,
if (size > MAX_VALUE_LEN(inode))
return -E2BIG;
retry:
error = read_all_xattrs(inode, ifolio, &base_addr);
error = read_all_xattrs(inode, ientry, &base_addr);
if (error)
return error;
@ -773,7 +774,7 @@ static int __f2fs_setxattr(struct inode *inode, int index,
*(u32 *)((u8 *)last + newsize) = 0;
}
error = write_all_xattrs(inode, new_hsize, base_addr, ifolio);
error = write_all_xattrs(inode, new_hsize, base_addr, ientry);
if (error)
goto exit;
@ -807,7 +808,7 @@ static int __f2fs_setxattr(struct inode *inode, int index,
int f2fs_setxattr(struct inode *inode, int index, const char *name,
const void *value, size_t size,
struct folio *ifolio, int flags)
struct f2fs_cached_block *ientry, int flags)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_lock_context lc;
@ -823,9 +824,9 @@ int f2fs_setxattr(struct inode *inode, int index, const char *name,
return err;
/* this case is only from f2fs_init_inode_metadata */
if (ifolio)
if (ientry)
return __f2fs_setxattr(inode, index, name, value,
size, ifolio, flags);
size, ientry, flags);
f2fs_balance_fs(sbi, true);
f2fs_lock_op(sbi, &lc);
@ -848,4 +849,4 @@ int __init f2fs_init_xattr_cache(void)
void f2fs_destroy_xattr_cache(void)
{
kmem_cache_destroy(inline_xattr_slab);
}
}

23
fs/f2fs/xattr.h
View File

@ -71,21 +71,22 @@ struct f2fs_xattr_entry {
for (entry = XATTR_FIRST_ENTRY(addr);\
!IS_XATTR_LAST_ENTRY(entry);\
entry = XATTR_NEXT_ENTRY(entry))
#define VALID_XATTR_BLOCK_SIZE (PAGE_SIZE - sizeof(struct node_footer))
#define VALID_XATTR_BLOCK_SIZE(i) (i_blocksize(i) - \
sizeof(struct node_footer))
#define XATTR_PADDING_SIZE (sizeof(__u32))
#define XATTR_SIZE(i) ((F2FS_I(i)->i_xattr_nid ? \
VALID_XATTR_BLOCK_SIZE : 0) + \
VALID_XATTR_BLOCK_SIZE(i) : 0) + \
(inline_xattr_size(i)))
#define MIN_OFFSET(i) XATTR_ALIGN(inline_xattr_size(i) + \
VALID_XATTR_BLOCK_SIZE)
VALID_XATTR_BLOCK_SIZE(i))
#define MAX_VALUE_LEN(i) (MIN_OFFSET(i) - \
sizeof(struct f2fs_xattr_header) - \
sizeof(struct f2fs_xattr_entry))
#define MIN_INLINE_XATTR_SIZE (sizeof(struct f2fs_xattr_header) / sizeof(__le32))
#define MAX_INLINE_XATTR_SIZE \
(DEF_ADDRS_PER_INODE - \
#define MAX_INLINE_XATTR_SIZE(blocksize) \
(F2FS_DEF_ADDRS_PER_INODE(blocksize) - \
F2FS_TOTAL_EXTRA_ATTR_SIZE / sizeof(__le32) - \
DEF_INLINE_RESERVED_SIZE - \
MIN_INLINE_DENTRY_SIZE / sizeof(__le32))
@ -130,9 +131,9 @@ extern const struct xattr_handler f2fs_xattr_security_handler;
extern const struct xattr_handler * const f2fs_xattr_handlers[];
int f2fs_setxattr(struct inode *, int, const char *, const void *,
size_t, struct folio *, int);
size_t, struct f2fs_cached_block *, int);
int f2fs_getxattr(struct inode *, int, const char *, void *,
size_t, struct folio *);
size_t, struct f2fs_cached_block *);
ssize_t f2fs_listxattr(struct dentry *, char *, size_t);
int __init f2fs_init_xattr_cache(void);
void f2fs_destroy_xattr_cache(void);
@ -142,13 +143,13 @@ void f2fs_destroy_xattr_cache(void);
#define f2fs_listxattr NULL
static inline int f2fs_setxattr(struct inode *inode, int index,
const char *name, const void *value, size_t size,
struct folio *folio, int flags)
struct f2fs_cached_block *ientry, int flags)
{
return -EOPNOTSUPP;
}
static inline int f2fs_getxattr(struct inode *inode, int index,
const char *name, void *buffer,
size_t buffer_size, struct folio *dfolio)
size_t buffer_size, struct f2fs_cached_block *ientry)
{
return -EOPNOTSUPP;
}
@ -158,10 +159,10 @@ static inline void f2fs_destroy_xattr_cache(void) { }
#ifdef CONFIG_F2FS_FS_SECURITY
int f2fs_init_security(struct inode *, struct inode *,
const struct qstr *, struct folio *);
const struct qstr *, struct f2fs_cached_block *);
#else
static inline int f2fs_init_security(struct inode *inode, struct inode *dir,
const struct qstr *qstr, struct folio *ifolio)
const struct qstr *qstr, struct f2fs_cached_block *ientry)
{
return 0;
}

140
include/linux/f2fs_fs.h
View File

@ -14,9 +14,9 @@
#define F2FS_SUPER_OFFSET 1024 /* byte-size offset */
#define F2FS_MIN_LOG_SECTOR_SIZE 9 /* 9 bits for 512 bytes */
#define F2FS_MAX_LOG_SECTOR_SIZE PAGE_SHIFT /* Max is Block Size */
#define F2FS_LOG_SECTORS_PER_BLOCK (PAGE_SHIFT - 9) /* log number for sector/blk */
#define F2FS_BLKSIZE PAGE_SIZE /* support only block == page */
#define F2FS_BLKSIZE_BITS PAGE_SHIFT /* bits for F2FS_BLKSIZE */
#define F2FS_MIN_LOG_BLOCKSIZE 12
#define F2FS_MIN_BLKSIZE 4096UL
#define F2FS_MAX_BLKSIZE PAGE_SIZE
#define F2FS_MAX_EXTENSION 64 /* # of extension entries */
#define F2FS_EXTENSION_LEN 8 /* max size of extension */
@ -24,19 +24,25 @@
#define NEW_ADDR ((block_t)-1) /* used as block_t addresses */
#define COMPRESS_ADDR ((block_t)-2) /* used as compressed data flag */
#define F2FS_BLKSIZE_MASK (F2FS_BLKSIZE - 1)
#define F2FS_BYTES_TO_BLK(bytes) ((unsigned long long)(bytes) >> F2FS_BLKSIZE_BITS)
#define F2FS_BLK_TO_BYTES(blk) ((unsigned long long)(blk) << F2FS_BLKSIZE_BITS)
#define F2FS_BLK_END_BYTES(blk) (F2FS_BLK_TO_BYTES(blk + 1) - 1)
#define F2FS_BLK_ALIGN(x) (F2FS_BYTES_TO_BLK((x) + F2FS_BLKSIZE - 1))
#define F2FS_BLKSIZE(sbi) ((sbi)->blocksize)
#define F2FS_BLKSIZE_BITS(sbi) ((sbi)->log_blocksize)
#define F2FS_BLKSIZE_MASK(sbi) (F2FS_BLKSIZE(sbi) - 1)
#define F2FS_LOG_SECTORS_PER_BLOCK(sbi) (F2FS_BLKSIZE_BITS(sbi) - 9)
#define F2FS_BLKS_PER_PAGE(sbi) (PAGE_SIZE / F2FS_BLKSIZE(sbi))
#define F2FS_BYTES_TO_BLK(sbi, bytes) \
((unsigned long long)(bytes) >> F2FS_BLKSIZE_BITS(sbi))
#define F2FS_BLK_TO_BYTES(sbi, blk) \
((unsigned long long)(blk) << F2FS_BLKSIZE_BITS(sbi))
#define F2FS_BLK_END_BYTES(sbi, blk) \
(F2FS_BLK_TO_BYTES(sbi, (blk) + 1) - 1)
#define F2FS_BLK_ALIGN(sbi, bytes) \
F2FS_BYTES_TO_BLK(sbi, (unsigned long long)(bytes) + \
F2FS_BLKSIZE(sbi) - 1)
/* 0, 1(node nid), 2(meta nid) are reserved node id */
#define F2FS_RESERVED_NODE_NUM 3
#define F2FS_ROOT_INO(sbi) ((sbi)->root_ino_num)
#define F2FS_NODE_INO(sbi) ((sbi)->node_ino_num)
#define F2FS_META_INO(sbi) ((sbi)->meta_ino_num)
#define F2FS_COMPRESS_INO(sbi) (NM_I(sbi)->max_nid)
#define F2FS_MAX_QUOTAS 3
@ -214,20 +220,27 @@ struct f2fs_checkpoint {
unsigned char sit_nat_version_bitmap[];
} __packed;
#define CP_CHKSUM_OFFSET (F2FS_BLKSIZE - sizeof(__le32)) /* default chksum offset in checkpoint */
#define CP_MIN_CHKSUM_OFFSET \
(offsetof(struct f2fs_checkpoint, sit_nat_version_bitmap))
/*
* For orphan inode management
*
* The number of inode entries in an orphan block depends on the filesystem
* block size. Its exact on-disk layout is:
*
* 0 blocksize - 16 blocksize
* +--------------------------+--------------------------+
* | ino[0] ... ino[n - 1] | struct f2fs_orphan_footer |
* +--------------------------+--------------------------+
*
* n = (blocksize - sizeof(struct f2fs_orphan_footer)) / sizeof(__le32)
*/
#define F2FS_ORPHANS_PER_BLOCK ((F2FS_BLKSIZE - 4 * sizeof(__le32)) / sizeof(__le32))
#define GET_ORPHAN_BLOCKS(n) (((n) + F2FS_ORPHANS_PER_BLOCK - 1) / \
F2FS_ORPHANS_PER_BLOCK)
struct f2fs_orphan_block {
__le32 ino[F2FS_ORPHANS_PER_BLOCK]; /* inode numbers */
DECLARE_FLEX_ARRAY(__le32, ino);
} __packed;
struct f2fs_orphan_footer {
__le32 reserved; /* reserved */
__le16 blk_addr; /* block index in current CP */
__le16 blk_count; /* Number of orphan inode blocks in CP */
@ -260,26 +273,14 @@ struct node_footer {
} __packed;
/* Address Pointers in an Inode */
#define DEF_ADDRS_PER_INODE ((F2FS_BLKSIZE - OFFSET_OF_END_OF_I_EXT \
- SIZE_OF_I_NID \
- sizeof(struct node_footer)) / sizeof(__le32))
#define CUR_ADDRS_PER_INODE(inode) (DEF_ADDRS_PER_INODE - \
get_extra_isize(inode))
#define F2FS_DEF_ADDRS_PER_INODE(blocksize) \
(((blocksize) - OFFSET_OF_END_OF_I_EXT - SIZE_OF_I_NID - \
sizeof(struct node_footer)) / sizeof(__le32))
#define DEF_NIDS_PER_INODE 5 /* Node IDs in an Inode */
#define ADDRS_PER_INODE(inode) addrs_per_page(inode, true)
/* Address Pointers in a Direct Block */
#define DEF_ADDRS_PER_BLOCK ((F2FS_BLKSIZE - sizeof(struct node_footer)) / sizeof(__le32))
#define ADDRS_PER_BLOCK(inode) addrs_per_page(inode, false)
/* Node IDs in an Indirect Block */
#define NIDS_PER_BLOCK ((F2FS_BLKSIZE - sizeof(struct node_footer)) / sizeof(__le32))
#define ADDRS_PER_PAGE(folio, inode) (addrs_per_page(inode, IS_INODE(folio)))
#define NODE_DIR1_BLOCK (DEF_ADDRS_PER_INODE + 1)
#define NODE_DIR2_BLOCK (DEF_ADDRS_PER_INODE + 2)
#define NODE_IND1_BLOCK (DEF_ADDRS_PER_INODE + 3)
#define NODE_IND2_BLOCK (DEF_ADDRS_PER_INODE + 4)
#define NODE_DIND_BLOCK (DEF_ADDRS_PER_INODE + 5)
#define ADDRS_PER_PAGE(folio, inode) (addrs_per_page(inode, IS_INODE(F2FS_I_SB(inode), folio)))
#define F2FS_INLINE_XATTR 0x01 /* file inline xattr flag */
#define F2FS_INLINE_DATA 0x02 /* file inline data flag */
@ -339,18 +340,26 @@ struct f2fs_inode {
*/
__le32 i_extra_end[0]; /* for attribute size calculation */
} __packed;
__le32 i_addr[DEF_ADDRS_PER_INODE]; /* Pointers to data blocks */
DECLARE_FLEX_ARRAY(__le32, i_addr); /* data block pointers */
};
__le32 i_nid[DEF_NIDS_PER_INODE]; /* direct(2), indirect(2),
double_indirect(1) node id */
/*
* __le32 i_nid[DEF_NIDS_PER_INODE];
* direct(2), indirect(2), double_indirect(1) node IDs
*
* It is stored immediately before the node footer at the end of the
* filesystem block. Its offset depends on the filesystem block size, so
* locate it dynamically with F2FS_INODE_NIDS().
*/
} __packed;
struct direct_node {
__le32 addr[DEF_ADDRS_PER_BLOCK]; /* array of data block address */
/* The address count depends on the filesystem block size. */
DECLARE_FLEX_ARRAY(__le32, addr); /* array of data block address */
} __packed;
struct indirect_node {
__le32 nid[NIDS_PER_BLOCK]; /* array of data block address */
/* The node ID count depends on the filesystem block size. */
DECLARE_FLEX_ARRAY(__le32, nid); /* array of data block address */
} __packed;
enum {
@ -369,14 +378,18 @@ struct f2fs_node {
struct direct_node dn;
struct indirect_node in;
};
struct node_footer footer;
/*
* struct node_footer footer;
*
* It is stored at the end of the filesystem block, after the inode or
* direct/indirect node data. Its offset depends on the filesystem block
* size, so locate it dynamically with F2FS_NODE_FOOTER().
*/
} __packed;
/*
* For NAT entries
*/
#define NAT_ENTRY_PER_BLOCK (F2FS_BLKSIZE / sizeof(struct f2fs_nat_entry))
struct f2fs_nat_entry {
__u8 version; /* latest version of cached nat entry */
__le32 ino; /* inode number */
@ -384,7 +397,8 @@ struct f2fs_nat_entry {
} __packed;
struct f2fs_nat_block {
struct f2fs_nat_entry entries[NAT_ENTRY_PER_BLOCK];
/* The entry count depends on the filesystem block size. */
DECLARE_FLEX_ARRAY(struct f2fs_nat_entry, entries);
} __packed;
/*
@ -396,8 +410,6 @@ struct f2fs_nat_block {
* Not allow to change this.
*/
#define SIT_VBLOCK_MAP_SIZE 64
#define SIT_ENTRY_PER_BLOCK (F2FS_BLKSIZE / sizeof(struct f2fs_sit_entry))
/*
* F2FS uses 4 bytes to represent block address. As a result, supported size of
* disk is 16 TB for a 4K page size and 64 TB for a 16K page size and it equals
@ -424,8 +436,13 @@ struct f2fs_sit_entry {
__le64 mtime; /* segment age for cleaning */
} __packed;
/*
* The on-disk SIT block is a filesystem-block-sized array of SIT entries.
* Its entry count depends on the filesystem block size, so it must be
* calculated by the caller rather than implied by this C structure.
*/
struct f2fs_sit_block {
struct f2fs_sit_entry entries[SIT_ENTRY_PER_BLOCK];
DECLARE_FLEX_ARRAY(struct f2fs_sit_entry, entries);
} __packed;
/*
@ -595,15 +612,7 @@ typedef __le32 f2fs_hash_t;
* dentry, when converting inline dentry we should handle this carefully.
*/
/* the number of dentry in a block */
#define NR_DENTRY_IN_BLOCK ((BITS_PER_BYTE * F2FS_BLKSIZE) / \
((SIZE_OF_DIR_ENTRY + F2FS_SLOT_LEN) * BITS_PER_BYTE + 1))
#define SIZE_OF_DIR_ENTRY 11 /* by byte */
#define SIZE_OF_DENTRY_BITMAP ((NR_DENTRY_IN_BLOCK + BITS_PER_BYTE - 1) / \
BITS_PER_BYTE)
#define SIZE_OF_RESERVED (F2FS_BLKSIZE - ((SIZE_OF_DIR_ENTRY + \
F2FS_SLOT_LEN) * \
NR_DENTRY_IN_BLOCK + SIZE_OF_DENTRY_BITMAP))
#define MIN_INLINE_DENTRY_SIZE 40 /* just include '.' and '..' entries */
/* One directory entry slot representing F2FS_SLOT_LEN-sized file name */
@ -614,14 +623,21 @@ struct f2fs_dir_entry {
__u8 file_type; /* file type */
} __packed;
/* Block-sized directory entry block */
struct f2fs_dentry_block {
/* validity bitmap for directory entries in each block */
__u8 dentry_bitmap[SIZE_OF_DENTRY_BITMAP];
__u8 reserved[SIZE_OF_RESERVED];
struct f2fs_dir_entry dentry[NR_DENTRY_IN_BLOCK];
__u8 filename[NR_DENTRY_IN_BLOCK][F2FS_SLOT_LEN];
} __packed;
/*
* A dentry block is laid out as follows, where the number of entries and all
* offsets are determined by the filesystem block size at runtime:
*
* 0 blocksize
* +--------+----------+-------------------+-----------------------+
* | bitmap | reserved | dir_entry[entries]| filename[entries][8] |
* +--------+----------+-------------------+-----------------------+
*
* entries = (BITS_PER_BYTE * blocksize) /
* ((SIZE_OF_DIR_ENTRY + F2FS_SLOT_LEN) * BITS_PER_BYTE + 1)
* bitmap_size = DIV_ROUND_UP(entries, BITS_PER_BYTE)
* reserved_size = blocksize - bitmap_size -
* (SIZE_OF_DIR_ENTRY + F2FS_SLOT_LEN) * entries
*/
#define F2FS_DEF_PROJID 0 /* default project ID */

71
include/trace/events/f2fs.h
View File

@ -1430,6 +1430,50 @@ DEFINE_EVENT(f2fs__folio, f2fs_set_page_dirty,
TP_ARGS(folio, type)
);
DECLARE_EVENT_CLASS(f2fs__cached_block,
TP_PROTO(struct f2fs_cached_block *block, int type),
TP_ARGS(block, type),
TP_STRUCT__entry(
__field(dev_t, dev)
__field(pgoff_t, index)
__field(int, type)
__field(int, dirty)
__field(int, uptodate)
),
TP_fast_assign(
__entry->dev = block->cache->sbi->sb->s_dev;
__entry->index = block->index;
__entry->type = type;
__entry->dirty = f2fs_cache_test_dirty(block);
__entry->uptodate = f2fs_cache_test_uptodate(block);
),
TP_printk("dev = (%d,%d), %s, index = %lu, dirty = %d, uptodate = %d",
show_dev(__entry->dev),
show_block_type(__entry->type),
(unsigned long)__entry->index,
__entry->dirty,
__entry->uptodate)
);
DEFINE_EVENT(f2fs__cached_block, f2fs_write_cache,
TP_PROTO(struct f2fs_cached_block *block, int type),
TP_ARGS(block, type)
);
DEFINE_EVENT(f2fs__cached_block, f2fs_cache_set_dirty,
TP_PROTO(struct f2fs_cached_block *block, int type),
TP_ARGS(block, type)
);
TRACE_EVENT(f2fs_replace_atomic_write_block,
TP_PROTO(struct inode *inode, struct inode *cow_inode, pgoff_t index,
@ -1574,6 +1618,33 @@ TRACE_EVENT(f2fs_writepages,
__entry->for_sync)
);
TRACE_EVENT(f2fs_write_caches,
TP_PROTO(struct f2fs_sb_info *sbi, long nr_to_write, long nwritten, int type),
TP_ARGS(sbi, nr_to_write, nwritten, type),
TP_STRUCT__entry(
__field(dev_t, dev)
__field(long, nr_to_write)
__field(long, nwritten)
__field(int, type)
),
TP_fast_assign(
__entry->dev = sbi->sb->s_dev;
__entry->nr_to_write = nr_to_write;
__entry->nwritten = nwritten;
__entry->type = type;
),
TP_printk("dev = (%d,%d), %s, nr_to_write = %ld, nwritten = %ld",
show_dev(__entry->dev),
show_block_type(__entry->type),
__entry->nr_to_write,
__entry->nwritten)
);
TRACE_EVENT(f2fs_readpages,
TP_PROTO(struct inode *inode, pgoff_t start, unsigned int nrpage),